2024-09-16T11:04:40.796 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T11:04:40.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T11:04:40.899 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: sangadi@redhat.com first_in_suite: false job_id: '7907167' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGqAFr/poL95sO4gd9TCrLiYuCVqFzL5efo5srdCOpECOtvr6+2wExCxiQcB5UD5JOjszyCNZqaCh2dYXBmp8Y= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIn6j46Ue+ntpffguTLbpcQNYJduNZD9odWBx7pmD0KsupCXWd35E+7vLdXpX1BFhsiUSaYof4nIDQ6cVd+aL2M= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_10:44:09 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T11:04:40.900 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-16T11:04:40.900 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-16T11:04:40.901 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T11:04:40.902 INFO:teuthology.task.internal:Checking packages... 2024-09-16T11:04:40.923 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-16T11:04:40.924 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T11:04:40.924 INFO:teuthology.packaging:ref: None 2024-09-16T11:04:40.924 INFO:teuthology.packaging:tag: None 2024-09-16T11:04:40.924 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T11:04:40.924 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:04:40.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T11:04:41.078 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-16T11:04:41.080 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T11:04:41.088 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T11:04:41.088 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T11:04:41.093 INFO:teuthology.task.internal:Saving configuration 2024-09-16T11:04:41.111 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T11:04:41.118 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T11:04:41.142 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:59:22.882287', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGqAFr/poL95sO4gd9TCrLiYuCVqFzL5efo5srdCOpECOtvr6+2wExCxiQcB5UD5JOjszyCNZqaCh2dYXBmp8Y='} 2024-09-16T11:04:41.166 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:59:22.880831', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIn6j46Ue+ntpffguTLbpcQNYJduNZD9odWBx7pmD0KsupCXWd35E+7vLdXpX1BFhsiUSaYof4nIDQ6cVd+aL2M='} 2024-09-16T11:04:41.166 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T11:04:41.171 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-16T11:04:41.171 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-16T11:04:41.171 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T11:04:41.232 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd11cb6fac0>, signals=[15]) 2024-09-16T11:04:41.232 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T11:04:41.237 INFO:teuthology.task.internal:Opening connections... 2024-09-16T11:04:41.238 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:04:41.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:04:41.315 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:04:41.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:04:41.389 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T11:04:41.396 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-09-16T11:04:41.415 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-09-16T11:04:41.415 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-09-16T11:04:41.473 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-09-16T11:04:41.473 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-09-16T11:04:41.473 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-09-16T11:04:41.473 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-09-16T11:04:41.473 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T11:04:41.474 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T11:04:41.475 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-09-16T11:04:41.502 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-09-16T11:04:41.521 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-09-16T11:04:41.521 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T11:04:41.579 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-09-16T11:04:41.580 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T11:04:41.580 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T11:04:41.580 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T11:04:41.580 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-09-16T11:04:41.603 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T11:04:41.611 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T11:04:41.616 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T11:04:41.616 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-09-16T11:04:41.619 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-09-16T11:04:41.635 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T11:04:41.640 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T11:04:41.641 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-09-16T11:04:41.676 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-09-16T11:04:41.706 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T11:04:41.761 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T11:04:41.775 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T11:04:41.775 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T11:04:41.775 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T11:04:41.776 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T11:04:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:04:41.792 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-16T11:04:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:04:41.794 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-16T11:04:41.851 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-503.el9.x86_64 2024-09-16T11:04:41.851 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-503.el9.x86_64 2024-09-16T11:04:41.851 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-09-16T11:04:41.852 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-503.el9.x86_64 2024-09-16T11:04:41.853 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-503.el9.x86_64 2024-09-16T11:04:41.853 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-09-16T11:04:45.136 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.2 MB 00:01 2024-09-16T11:04:46.456 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.2 MB 00:02 2024-09-16T11:04:47.421 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-16T11:04:48.224 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-09-16T11:04:51.788 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-16T11:04:53.305 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-16T11:04:54.160 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-16T11:04:54.458 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-16T11:04:55.884 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-16T11:04:56.755 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T11:04:59.824 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T11:05:01.922 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T11:05:01.922 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T11:05:02.023 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:05:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-09-16T11:05:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:05:02.037 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 79 M 2024-09-16T11:05:02.037 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 126 M 2024-09-16T11:05:02.037 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:05:02.227 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T11:05:02.384 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 324 kB/s | 23 kB 00:00 2024-09-16T11:05:02.792 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-16T11:05:03.501 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-16T11:05:03.785 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-16T11:05:03.785 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:05:03.785 INFO:teuthology.orchestra.run.smithi001.stdout:Total 45 MB/s | 79 MB 00:01 2024-09-16T11:05:04.259 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T11:05:04.259 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T11:05:04.341 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:05:04.353 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:05:04.353 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-16T11:05:04.353 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:05:04.353 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-16T11:05:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:05:04.355 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 79 M 2024-09-16T11:05:04.355 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 126 M 2024-09-16T11:05:04.355 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:05:04.492 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:05:04.495 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 191 kB/s | 23 kB 00:00 2024-09-16T11:05:04.855 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:05:04.855 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:05:05.826 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:05:05.826 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:05:08.391 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:05:09.053 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:05:09.134 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:05:11.254 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-16T11:05:11.906 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:05:12.038 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:05:17.159 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:05:17.197 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:05:17.803 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 2.1 MB/s | 29 MB 00:13 2024-09-16T11:05:20.629 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.1 MB/s | 35 MB 00:16 2024-09-16T11:05:20.629 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:05:20.629 INFO:teuthology.orchestra.run.smithi103.stdout:Total 4.8 MB/s | 79 MB 00:16 2024-09-16T11:05:21.316 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:05:21.690 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:05:21.690 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:05:22.340 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:05:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:05:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:05:25.122 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:05:25.823 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:05:25.923 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:05:28.732 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:05:28.875 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:05:33.750 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:05:33.783 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:05:38.647 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:01.321 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:01.362 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:01.753 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:01.753 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:06:01.753 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:06:01.753 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:06:02.619 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:06:02.942 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:06:03.030 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:06:03.030 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:06:03.031 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T11:06:03.031 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T11:06:03.031 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T11:06:03.031 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-16T11:06:03.031 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T11:06:03.031 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:06:03.090 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:06:03.090 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:06:03.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-09-16T11:06:03.128 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-09-16T11:06:03.128 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 82.el9 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8274670 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-09-16T11:06:03.129 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:06:03.130 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T11:06:03.131 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T11:06:03.131 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-505.el9.x86_64 2024-09-16T11:06:03.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T11:06:03.757 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-09-16T11:06:05.186 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T11:06:05.210 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-09-16T11:06:05.212 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-16T11:06:05.228 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.KDaU5pXEI7 2024-09-16T11:06:05.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KDaU5pXEI7 2024-09-16T11:06:05.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.KDaU5pXEI7 2024-09-16T11:06:05.447 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.KDaU5pXEI7 is 6KB 2024-09-16T11:06:05.497 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.KDaU5pXEI7 2024-09-16T11:06:05.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T11:06:05.578 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T11:06:05.578 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T11:06:05.578 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T11:06:05.578 INFO:teuthology.orchestra.run.smithi001.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T11:06:05.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T11:06:05.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-09-16T11:06:16.967 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:17.013 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:06:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:06:17.363 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:06:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:06:18.190 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:06:18.540 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:06:18.623 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:06:18.623 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:06:18.623 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T11:06:18.623 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T11:06:18.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T11:06:18.624 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-16T11:06:18.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T11:06:18.624 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:06:18.683 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:06:18.684 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:06:18.684 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-09-16T11:06:18.715 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-09-16T11:06:18.715 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-09-16T11:06:18.715 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 82.el9 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8274670 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T11:06:18.716 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:06:18.717 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T11:06:18.718 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T11:06:18.718 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-505.el9.x86_64 2024-09-16T11:06:18.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T11:06:19.418 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-09-16T11:06:20.782 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T11:06:20.818 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-09-16T11:06:20.819 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-16T11:06:20.835 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.NhB8KxwhmP 2024-09-16T11:06:20.835 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NhB8KxwhmP 2024-09-16T11:06:20.902 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.NhB8KxwhmP 2024-09-16T11:06:21.046 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.NhB8KxwhmP is 6KB 2024-09-16T11:06:21.094 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.NhB8KxwhmP 2024-09-16T11:06:21.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T11:06:21.171 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T11:06:21.172 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T11:06:21.172 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T11:06:21.172 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T11:06:21.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T11:06:21.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-09-16T11:06:35.749 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-16T11:06:35.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-16T11:06:35.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:06:51.353 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-16T11:06:51.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:06:51.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:06:54.145 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-09-16T11:07:03.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-16T11:07:03.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:07:09.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-16T11:07:18.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:07:18.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:07:21.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-16T11:07:33.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:07:33.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:07:36.893 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-16T11:07:51.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:07:51.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:08:03.168 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T11:08:15.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-16T11:08:15.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:08:15.539 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-16T11:08:15.930 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-16T11:08:15.930 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T11:08:15.930 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-09-16T11:08:15.946 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-505.el9.x86_64 2024-09-16T11:08:15.947 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T11:08:15.947 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T11:08:15.947 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T11:08:16.948 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-16T11:08:16.948 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T11:08:17.033 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-09-16T11:08:17.058 DEBUG:teuthology.parallel:result is None 2024-09-16T11:08:23.886 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-16T11:08:24.227 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:08:24.227 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T11:08:24.227 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-16T11:08:24.243 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-505.el9.x86_64 2024-09-16T11:08:24.243 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T11:08:24.243 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T11:08:24.243 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T11:08:25.244 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-16T11:08:25.244 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T11:08:25.317 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-09-16T11:08:25.345 DEBUG:teuthology.parallel:result is None 2024-09-16T11:08:25.345 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T11:08:25.352 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T11:08:25.353 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T11:08:25.355 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T11:08:25.374 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T11:08:25.382 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T11:08:25.394 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T11:08:25.395 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T11:08:25.414 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T11:08:25.450 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T11:08:25.460 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T11:08:25.460 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T11:08:25.477 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T11:08:25.507 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:25.514 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:25.519 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:25.525 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:25.527 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T11:08:25.534 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T11:08:25.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T11:08:25.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T11:08:25.595 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T11:08:25.606 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T11:08:25.606 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T11:08:25.633 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T11:08:25.652 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:08:25.733 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:08:25.783 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:08:25.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:08:25.909 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:08:25.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T11:08:25.976 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:08:26.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:08:26.066 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:08:26.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:08:26.192 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:08:26.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T11:08:26.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-09-16T11:08:26.260 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-09-16T11:08:26.300 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:08:26.347 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:08:26.861 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T11:08:26.868 INFO:teuthology.task.internal:Starting timer... 2024-09-16T11:08:26.868 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T11:08:26.878 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T11:08:26.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-09-16T11:08:26.966 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-09-16T11:08:26.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-09-16T11:08:27.026 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-09-16T11:08:27.028 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T11:08:27.028 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-09-16T11:08:27.049 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-09-16T11:08:27.050 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-09-16T11:08:27.081 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-09-16T11:08:27.081 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-09-16T11:08:27.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:08:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:08:27.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:08:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:08:27.152 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T11:08:27.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-09-16T11:08:27.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-09-16T11:08:27.231 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T11:08:27.240 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T11:08:27.243 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T11:08:27.349 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T11:08:27.363 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T11:08:27.364 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T11:12:16.361 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-09-16T11:12:16.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-16T11:12:16.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:12:16.440 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-09-16T11:12:16.521 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-09-16T11:12:16.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:12:16.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:12:16.596 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-16T11:12:16.676 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-16T11:12:16.676 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T11:12:16.686 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T11:12:16.687 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T11:12:16.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:12:16.690 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T11:12:16.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:12:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T11:12:16.736 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T11:12:16.744 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T11:12:16.758 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T11:12:16.762 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-09-16T11:12:16.774 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-09-16T11:12:16.782 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-09-16T11:12:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T11:12:16.793 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-09-16T11:12:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T11:12:16.807 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T11:12:16.821 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T11:12:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:12:16.865 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:12:16.865 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-09-16T11:12:16.866 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.866 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.866 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.866 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.877 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:12:16.880 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:12:16.880 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-16T11:12:16.880 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.881 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:12:16.882 INFO:teuthology.run_tasks:Running task install... 2024-09-16T11:12:16.892 DEBUG:teuthology.task.install:project ceph 2024-09-16T11:12:16.892 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T11:12:16.892 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-16T11:12:16.892 INFO:teuthology.task.install:Using flavor: default 2024-09-16T11:12:16.898 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T11:12:16.898 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-16T11:12:16.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T11:12:16.898 DEBUG:teuthology.orchestra.run.smithi103:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T11:12:17.481 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:50 ago on Mon 16 Sep 2024 11:11:27 AM UTC. 2024-09-16T11:12:17.489 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:47 ago on Mon 16 Sep 2024 11:11:30 AM UTC. 2024-09-16T11:12:17.569 INFO:teuthology.orchestra.run.smithi103.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T11:12:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T11:12:17.612 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:12:17.613 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:12:17.613 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:12:17.624 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:12:17.625 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:12:17.625 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:12:17.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo dnf -y copr enable ceph/el9 2024-09-16T11:12:17.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo dnf -y copr enable ceph/el9 2024-09-16T11:12:17.844 INFO:teuthology.orchestra.run.smithi103.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:of the main distribution, and quality may vary. 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:, 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:and packages are not held to any quality or security level. 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T11:12:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T11:12:17.870 INFO:teuthology.orchestra.run.smithi001.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T11:12:17.870 INFO:teuthology.orchestra.run.smithi001.stderr:of the main distribution, and quality may vary. 2024-09-16T11:12:17.870 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:, 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:and packages are not held to any quality or security level. 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T11:12:17.871 INFO:teuthology.orchestra.run.smithi001.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T11:12:17.896 INFO:teuthology.orchestra.run.smithi103.stdout:Repository successfully enabled. 2024-09-16T11:12:17.925 INFO:teuthology.orchestra.run.smithi001.stdout:Repository successfully enabled. 2024-09-16T11:12:17.931 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T11:12:17.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:12:17.965 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T11:12:17.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:12:18.083 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T11:12:18.084 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T11:12:18.108 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T11:12:18.108 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T11:12:18.198 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T11:12:18.198 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:12:18.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T11:12:18.217 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T11:12:18.217 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:12:18.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T11:12:18.223 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T11:12:18.224 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T11:12:18.243 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T11:12:18.244 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T11:12:18.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T11:12:18.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T11:12:18.363 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T11:12:18.387 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T11:12:18.428 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-09-16T11:12:18.429 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-09-16T11:12:18.452 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-09-16T11:12:18.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-09-16T11:12:18.663 INFO:teuthology.orchestra.run.smithi103.stdout:45 files removed 2024-09-16T11:12:18.687 INFO:teuthology.orchestra.run.smithi001.stdout:45 files removed 2024-09-16T11:12:18.689 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-09-16T11:12:18.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-09-16T11:12:19.238 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 291 kB/s | 77 kB 00:00 2024-09-16T11:12:19.293 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 276 kB/s | 77 kB 00:00 2024-09-16T11:12:19.522 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-16T11:12:19.594 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-09-16T11:12:19.798 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-16T11:12:19.870 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-16T11:12:19.965 INFO:teuthology.orchestra.run.smithi103.stdout:Copr repo for el9 owned by ceph 58 kB/s | 7.6 kB 00:00 2024-09-16T11:12:20.028 INFO:teuthology.orchestra.run.smithi001.stdout:Copr repo for el9 owned by ceph 62 kB/s | 7.6 kB 00:00 2024-09-16T11:12:20.512 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-16T11:12:20.576 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-16T11:12:22.575 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-16T11:12:22.772 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-16T11:12:27.210 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.5 MB 00:00 2024-09-16T11:12:27.532 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.5 MB 00:00 2024-09-16T11:12:28.956 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 118 kB/s | 19 kB 00:00 2024-09-16T11:12:29.294 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 19 kB 00:00 2024-09-16T11:12:29.786 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-16T11:12:30.223 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-16T11:12:35.519 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T11:12:35.729 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T11:12:38.122 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T11:12:38.135 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T11:12:38.136 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:12:38.137 INFO:teuthology.orchestra.run.smithi001.stdout:Install 22 Packages 2024-09-16T11:12:38.138 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-09-16T11:12:38.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:12:38.138 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 58 M 2024-09-16T11:12:38.138 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:12:38.179 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:12:38.186 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:12:38.187 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T11:12:38.188 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout:Install 22 Packages 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-09-16T11:12:38.189 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:12:38.190 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 58 M 2024-09-16T11:12:38.190 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:12:39.181 INFO:teuthology.orchestra.run.smithi103.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 7.6 MB/s | 5.0 MB 00:00 2024-09-16T11:12:39.223 INFO:teuthology.orchestra.run.smithi103.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 605 kB/s | 25 kB 00:00 2024-09-16T11:12:39.381 INFO:teuthology.orchestra.run.smithi103.stdout:(3/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 17 MB/s | 14 MB 00:00 2024-09-16T11:12:39.415 INFO:teuthology.orchestra.run.smithi103.stdout:(4/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 3.2 MB/s | 637 kB 00:00 2024-09-16T11:12:39.440 INFO:teuthology.orchestra.run.smithi103.stdout:(5/24): libradosstriper1-17.2.7-1643.gd75f2382. 7.7 MB/s | 458 kB 00:00 2024-09-16T11:12:39.499 INFO:teuthology.orchestra.run.smithi103.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 775 kB/s | 45 kB 00:00 2024-09-16T11:12:39.591 INFO:teuthology.orchestra.run.smithi103.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 1.3 MB/s | 120 kB 00:00 2024-09-16T11:12:39.691 INFO:teuthology.orchestra.run.smithi103.stdout:(8/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 18 MB/s | 4.9 MB 00:00 2024-09-16T11:12:39.716 INFO:teuthology.orchestra.run.smithi103.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.2 MB/s | 156 kB 00:00 2024-09-16T11:12:39.741 INFO:teuthology.orchestra.run.smithi103.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 6.3 MB/s | 320 kB 00:00 2024-09-16T11:12:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-16T11:12:39.792 INFO:teuthology.orchestra.run.smithi103.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-16T11:12:39.962 INFO:teuthology.orchestra.run.smithi001.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 4.5 MB/s | 5.0 MB 00:01 2024-09-16T11:12:40.009 INFO:teuthology.orchestra.run.smithi103.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 15 MB/s | 22 MB 00:01 2024-09-16T11:12:40.011 INFO:teuthology.orchestra.run.smithi001.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 508 kB/s | 25 kB 00:00 2024-09-16T11:12:40.162 INFO:teuthology.orchestra.run.smithi001.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 4.2 MB/s | 637 kB 00:00 2024-09-16T11:12:40.320 INFO:teuthology.orchestra.run.smithi001.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.8 MB/s | 458 kB 00:00 2024-09-16T11:12:40.509 INFO:teuthology.orchestra.run.smithi103.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-09-16T11:12:40.768 INFO:teuthology.orchestra.run.smithi103.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 161 kB/s | 160 kB 00:00 2024-09-16T11:12:40.885 INFO:teuthology.orchestra.run.smithi103.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 757 kB/s | 662 kB 00:00 2024-09-16T11:12:40.936 INFO:teuthology.orchestra.run.smithi103.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-16T11:12:40.961 INFO:teuthology.orchestra.run.smithi103.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T11:12:40.986 INFO:teuthology.orchestra.run.smithi103.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-16T11:12:41.037 INFO:teuthology.orchestra.run.smithi103.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 32 MB/s | 1.6 MB 00:00 2024-09-16T11:12:41.037 INFO:teuthology.orchestra.run.smithi001.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 6.9 MB/s | 4.9 MB 00:00 2024-09-16T11:12:41.113 INFO:teuthology.orchestra.run.smithi001.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 605 kB/s | 45 kB 00:00 2024-09-16T11:12:41.180 INFO:teuthology.orchestra.run.smithi001.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 1.8 MB/s | 120 kB 00:00 2024-09-16T11:12:41.280 INFO:teuthology.orchestra.run.smithi001.stdout:(8/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.5 MB/s | 156 kB 00:00 2024-09-16T11:12:41.403 INFO:teuthology.orchestra.run.smithi103.stdout:(21/24): python3-prettytable-0.7.2-27.el9.noarc 67 kB/s | 42 kB 00:00 2024-09-16T11:12:41.429 INFO:teuthology.orchestra.run.smithi103.stdout:(22/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 318 kB/s | 292 kB 00:00 2024-09-16T11:12:41.487 INFO:teuthology.orchestra.run.smithi103.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-09-16T11:12:41.497 INFO:teuthology.orchestra.run.smithi001.stdout:(9/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 5.5 MB/s | 14 MB 00:02 2024-09-16T11:12:41.531 INFO:teuthology.orchestra.run.smithi001.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 1.2 MB/s | 320 kB 00:00 2024-09-16T11:12:41.756 INFO:teuthology.orchestra.run.smithi001.stdout:(11/24): ceph-common-17.2.7-1643.gd75f2382.el9. 7.8 MB/s | 22 MB 00:02 2024-09-16T11:12:41.781 INFO:teuthology.orchestra.run.smithi001.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 398 kB/s | 100 kB 00:00 2024-09-16T11:12:41.815 INFO:teuthology.orchestra.run.smithi001.stdout:(13/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 929 kB/s | 295 kB 00:00 2024-09-16T11:12:41.890 INFO:teuthology.orchestra.run.smithi001.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-16T11:12:41.895 INFO:teuthology.orchestra.run.smithi103.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 6.0 MB/s | 2.9 MB 00:00 2024-09-16T11:12:41.900 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:12:41.900 INFO:teuthology.orchestra.run.smithi103.stdout:Total 16 MB/s | 58 MB 00:03 2024-09-16T11:12:41.915 INFO:teuthology.orchestra.run.smithi001.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2024-09-16T11:12:41.957 INFO:teuthology.orchestra.run.smithi001.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-16T11:12:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:12:41.999 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:12:41.999 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:12:41.999 INFO:teuthology.orchestra.run.smithi001.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 506 kB/s | 42 kB 00:00 2024-09-16T11:12:42.024 INFO:teuthology.orchestra.run.smithi001.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-16T11:12:42.083 INFO:teuthology.orchestra.run.smithi001.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-09-16T11:12:42.116 INFO:teuthology.orchestra.run.smithi001.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 887 kB/s | 67 kB 00:00 2024-09-16T11:12:42.142 INFO:teuthology.orchestra.run.smithi001.stdout:(21/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.6 MB/s | 308 kB 00:00 2024-09-16T11:12:42.192 INFO:teuthology.orchestra.run.smithi001.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T11:12:42.568 INFO:teuthology.orchestra.run.smithi001.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 6.9 MB/s | 2.9 MB 00:00 2024-09-16T11:12:42.630 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:12:42.631 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:12:42.793 INFO:teuthology.orchestra.run.smithi001.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 4.6 MB/s | 3.1 MB 00:00 2024-09-16T11:12:42.798 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:12:42.798 INFO:teuthology.orchestra.run.smithi001.stdout:Total 12 MB/s | 58 MB 00:04 2024-09-16T11:12:42.864 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:12:42.902 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:12:42.903 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:12:43.565 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:12:43.565 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:12:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:12:44.242 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T11:12:44.307 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:12:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:12:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:12:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:12:44.589 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:12:44.643 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T11:12:44.672 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T11:12:44.800 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:12:45.109 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T11:12:45.136 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:12:45.199 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:12:45.203 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T11:12:45.236 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T11:12:45.253 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:12:45.279 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T11:12:45.316 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:12:45.330 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:12:45.392 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:12:45.410 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:12:45.414 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:12:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:12:45.472 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:12:45.510 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:12:45.511 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T11:12:45.547 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T11:12:45.556 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T11:12:45.581 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T11:12:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T11:12:45.875 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T11:12:45.919 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:12:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:12:46.047 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T11:12:46.074 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:12:46.082 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T11:12:46.113 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T11:12:46.154 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:12:46.190 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T11:12:46.234 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T11:12:46.280 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:12:46.343 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:12:46.369 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:12:46.426 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:12:46.466 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T11:12:46.502 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T11:12:46.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T11:12:46.843 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T11:12:46.881 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:12:46.972 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:12:47.047 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T11:12:47.074 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T11:12:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:12:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:12:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:12:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:12:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:12:49.376 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T11:12:49.376 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T11:12:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:12:49.415 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:12:49.646 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:12:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:12:50.251 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:12:50.283 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:12:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:12:50.394 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T11:12:50.394 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T11:12:50.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:12:50.426 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:12:59.303 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:12:59.303 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-09-16T11:12:59.303 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-09-16T11:12:59.303 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-09-16T11:12:59.303 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-09-16T11:12:59.304 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-09-16T11:12:59.304 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-09-16T11:12:59.304 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-09-16T11:12:59.304 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:00.528 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:13:00.528 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-09-16T11:13:00.528 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-09-16T11:13:00.529 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-09-16T11:13:00.529 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-09-16T11:13:00.529 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-09-16T11:13:00.529 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-09-16T11:13:00.529 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-09-16T11:13:00.529 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:00.624 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:13:00.685 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:13:00.685 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:00.685 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T11:13:00.686 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:13:00.686 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:13:00.686 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:00.687 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:13:00.777 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:13:00.777 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:13:01.915 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:13:01.995 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:13:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T11:13:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:13:01.995 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:13:01.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:01.997 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:13:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:13:02.086 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:13:02.147 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:13:02.147 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T11:13:02.147 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:13:02.147 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T11:13:02.148 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T11:13:02.149 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T11:13:03.179 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:13:03.179 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:03.179 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-09-16T11:13:03.179 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.179 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.179 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.180 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:03.181 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:13:03.355 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T11:13:03.356 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T11:13:03.357 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T11:13:03.510 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-09-16T11:13:04.074 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:04.206 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:13:04.207 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:04.208 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-16T11:13:04.208 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:04.208 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 48 M 2024-09-16T11:13:04.208 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 193 M 2024-09-16T11:13:04.208 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:13:04.375 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-16T11:13:04.442 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 972 kB/s | 64 kB 00:00 2024-09-16T11:13:04.444 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:13:04.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:04.444 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T11:13:04.445 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:04.446 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:04.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-09-16T11:13:05.405 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:05.552 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T11:13:05.553 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:05.554 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 48 M 2024-09-16T11:13:05.555 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 193 M 2024-09-16T11:13:05.555 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:06.024 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 770 kB/s | 163 kB 00:00 2024-09-16T11:13:06.400 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-16T11:13:06.441 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 484 kB/s | 304 kB 00:00 2024-09-16T11:13:08.591 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 17 MB/s | 48 MB 00:02 2024-09-16T11:13:08.594 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:08.594 INFO:teuthology.orchestra.run.smithi001.stdout:Total 16 MB/s | 48 MB 00:03 2024-09-16T11:13:08.622 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:08.643 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:08.644 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:09.028 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:09.029 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:09.578 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:09.661 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T11:13:09.700 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T11:13:09.728 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:13:14.418 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:13:14.493 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:13:16.721 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:13:16.721 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T11:13:16.721 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T11:13:16.722 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T11:13:17.050 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T11:13:17.050 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:17.051 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:17.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-09-16T11:13:17.938 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:42 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:18.087 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T11:13:18.090 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T11:13:18.091 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T11:13:18.092 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout:Install 34 Packages 2024-09-16T11:13:18.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:18.094 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 27 M 2024-09-16T11:13:18.094 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 96 M 2024-09-16T11:13:18.094 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:18.591 INFO:teuthology.orchestra.run.smithi001.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-16T11:13:18.792 INFO:teuthology.orchestra.run.smithi001.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.2 MB/s | 2.1 MB 00:00 2024-09-16T11:13:18.842 INFO:teuthology.orchestra.run.smithi001.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.7 MB/s | 1.4 MB 00:00 2024-09-16T11:13:18.942 INFO:teuthology.orchestra.run.smithi001.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.3 MB/s | 240 kB 00:00 2024-09-16T11:13:19.126 INFO:teuthology.orchestra.run.smithi001.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T11:13:19.251 INFO:teuthology.orchestra.run.smithi001.stdout:(6/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 5.9 MB/s | 3.9 MB 00:00 2024-09-16T11:13:19.418 INFO:teuthology.orchestra.run.smithi001.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 640 kB/s | 106 kB 00:00 2024-09-16T11:13:19.594 INFO:teuthology.orchestra.run.smithi001.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 19 MB/s | 15 MB 00:00 2024-09-16T11:13:19.627 INFO:teuthology.orchestra.run.smithi001.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 2.4 MB/s | 1.2 MB 00:00 2024-09-16T11:13:19.652 INFO:teuthology.orchestra.run.smithi001.stdout:(10/34): python3-pycparser-2.20-6.el9.noarch.rp 578 kB/s | 135 kB 00:00 2024-09-16T11:13:19.699 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 11 kB/s | 163 kB 00:15 2024-09-16T11:13:19.710 INFO:teuthology.orchestra.run.smithi001.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-16T11:13:19.736 INFO:teuthology.orchestra.run.smithi001.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T11:13:19.761 INFO:teuthology.orchestra.run.smithi001.stdout:(13/34): python3-requests-2.25.1-8.el9.noarch.r 746 kB/s | 125 kB 00:00 2024-09-16T11:13:19.786 INFO:teuthology.orchestra.run.smithi001.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 460 kB/s | 35 kB 00:00 2024-09-16T11:13:19.812 INFO:teuthology.orchestra.run.smithi001.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-16T11:13:19.829 INFO:teuthology.orchestra.run.smithi001.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 759 kB/s | 51 kB 00:00 2024-09-16T11:13:19.845 INFO:teuthology.orchestra.run.smithi001.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 708 kB/s | 42 kB 00:00 2024-09-16T11:13:19.888 INFO:teuthology.orchestra.run.smithi001.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-09-16T11:13:19.904 INFO:teuthology.orchestra.run.smithi001.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 738 kB/s | 11 kB 00:00 2024-09-16T11:13:19.930 INFO:teuthology.orchestra.run.smithi001.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-16T11:13:19.946 INFO:teuthology.orchestra.run.smithi001.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-09-16T11:13:19.963 INFO:teuthology.orchestra.run.smithi001.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 694 kB/s | 23 kB 00:00 2024-09-16T11:13:19.981 INFO:teuthology.orchestra.run.smithi001.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 577 kB/s | 19 kB 00:00 2024-09-16T11:13:19.997 INFO:teuthology.orchestra.run.smithi001.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 585 kB/s | 20 kB 00:00 2024-09-16T11:13:20.023 INFO:teuthology.orchestra.run.smithi001.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-16T11:13:20.048 INFO:teuthology.orchestra.run.smithi001.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T11:13:20.082 INFO:teuthology.orchestra.run.smithi001.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-16T11:13:20.107 INFO:teuthology.orchestra.run.smithi001.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-16T11:13:20.132 INFO:teuthology.orchestra.run.smithi001.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T11:13:20.157 INFO:teuthology.orchestra.run.smithi001.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-16T11:13:20.183 INFO:teuthology.orchestra.run.smithi001.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-16T11:13:20.208 INFO:teuthology.orchestra.run.smithi001.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-16T11:13:20.233 INFO:teuthology.orchestra.run.smithi001.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T11:13:20.267 INFO:teuthology.orchestra.run.smithi001.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T11:13:20.272 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:20.272 INFO:teuthology.orchestra.run.smithi001.stdout:Total 12 MB/s | 27 MB 00:02 2024-09-16T11:13:20.410 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:20.452 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:20.452 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:20.877 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:20.877 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:21.033 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 48 MB 00:16 2024-09-16T11:13:21.035 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:21.035 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.9 MB/s | 48 MB 00:16 2024-09-16T11:13:21.062 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:13:21.081 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:13:21.081 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:13:21.447 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:13:21.447 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:13:21.970 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:13:22.051 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T11:13:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T11:13:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:13:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:22.220 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T11:13:22.253 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T11:13:22.285 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T11:13:22.382 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T11:13:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T11:13:22.505 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T11:13:22.548 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T11:13:22.577 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T11:13:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T11:13:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T11:13:22.695 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T11:13:22.738 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T11:13:22.772 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T11:13:22.821 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T11:13:22.861 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:13:22.938 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T11:13:23.090 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T11:13:23.175 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T11:13:23.235 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T11:13:23.272 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T11:13:23.331 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T11:13:23.395 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T11:13:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T11:13:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T11:13:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T11:13:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T11:13:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T11:13:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T11:13:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T11:13:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:13:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:13:24.205 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:24.205 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T11:13:24.205 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:13:24.205 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:13:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:13:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:13:25.622 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:25.622 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T11:13:25.622 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:13:25.623 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:13:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:26.019 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:13:26.045 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:13:26.045 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:26.045 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T11:13:26.045 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:13:26.045 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:13:26.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:26.253 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:13:26.279 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:13:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T11:13:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:13:26.279 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:13:26.279 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:26.345 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:13:26.572 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:13:26.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:13:28.000 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:13:28.000 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T11:13:28.000 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T11:13:28.001 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T11:13:28.002 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T11:13:28.003 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T11:13:28.003 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T11:13:28.003 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T11:13:28.003 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T11:13:28.003 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T11:13:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:13:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T11:13:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T11:13:28.945 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T11:13:29.014 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T11:13:29.014 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:29.014 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T11:13:29.015 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T11:13:29.016 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:29.017 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:29.244 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T11:13:29.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:29.244 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:13:29.244 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.245 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:29.245 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T11:13:29.245 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T11:13:29.245 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:29.245 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:29.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-09-16T11:13:29.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-09-16T11:13:29.973 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:30.065 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:13:30.101 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:30.109 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:30.110 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:13:30.111 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:30.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-09-16T11:13:30.237 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:30.239 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:30.239 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T11:13:30.240 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T11:13:30.241 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:30.242 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:13:30.243 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:30.243 INFO:teuthology.orchestra.run.smithi103.stdout:Install 34 Packages 2024-09-16T11:13:30.243 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:30.243 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 27 M 2024-09-16T11:13:30.243 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 96 M 2024-09-16T11:13:30.243 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:13:30.669 INFO:teuthology.orchestra.run.smithi103.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 54 kB/s | 6.5 kB 00:00 2024-09-16T11:13:30.762 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:30.844 INFO:teuthology.orchestra.run.smithi103.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.8 MB/s | 1.4 MB 00:00 2024-09-16T11:13:30.894 INFO:teuthology.orchestra.run.smithi103.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-09-16T11:13:30.900 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:30.900 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:30.900 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:13:30.900 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:30.900 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:30.901 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 83 k 2024-09-16T11:13:30.902 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 375 k 2024-09-16T11:13:30.902 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:30.986 INFO:teuthology.orchestra.run.smithi103.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-16T11:13:31.011 INFO:teuthology.orchestra.run.smithi103.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.0 MB/s | 240 kB 00:00 2024-09-16T11:13:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 268 kB/s | 83 kB 00:00 2024-09-16T11:13:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:31.210 INFO:teuthology.orchestra.run.smithi001.stdout:Total 267 kB/s | 83 kB 00:00 2024-09-16T11:13:31.211 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:31.216 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:31.218 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:31.218 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:31.253 INFO:teuthology.orchestra.run.smithi103.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 948 kB/s | 253 kB 00:00 2024-09-16T11:13:31.290 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:31.329 INFO:teuthology.orchestra.run.smithi103.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-16T11:13:31.496 INFO:teuthology.orchestra.run.smithi103.stdout:(8/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 24 MB/s | 15 MB 00:00 2024-09-16T11:13:31.512 INFO:teuthology.orchestra.run.smithi103.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 737 kB/s | 135 kB 00:00 2024-09-16T11:13:31.546 INFO:teuthology.orchestra.run.smithi103.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.3 MB/s | 1.2 MB 00:00 2024-09-16T11:13:31.580 INFO:teuthology.orchestra.run.smithi103.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-16T11:13:31.655 INFO:teuthology.orchestra.run.smithi103.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-16T11:13:31.680 INFO:teuthology.orchestra.run.smithi103.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-09-16T11:13:31.714 INFO:teuthology.orchestra.run.smithi103.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-16T11:13:31.739 INFO:teuthology.orchestra.run.smithi103.stdout:(15/34): python3-requests-2.25.1-8.el9.noarch.r 513 kB/s | 125 kB 00:00 2024-09-16T11:13:31.744 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:31.765 INFO:teuthology.orchestra.run.smithi103.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 605 kB/s | 51 kB 00:00 2024-09-16T11:13:31.790 INFO:teuthology.orchestra.run.smithi103.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-16T11:13:31.840 INFO:teuthology.orchestra.run.smithi103.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-16T11:13:31.841 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:31.865 INFO:teuthology.orchestra.run.smithi103.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-16T11:13:31.891 INFO:teuthology.orchestra.run.smithi103.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T11:13:31.916 INFO:teuthology.orchestra.run.smithi103.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-16T11:13:31.942 INFO:teuthology.orchestra.run.smithi103.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-16T11:13:31.967 INFO:teuthology.orchestra.run.smithi103.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-16T11:13:31.992 INFO:teuthology.orchestra.run.smithi103.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-16T11:13:32.009 INFO:teuthology.orchestra.run.smithi103.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-09-16T11:13:32.034 INFO:teuthology.orchestra.run.smithi103.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-16T11:13:32.060 INFO:teuthology.orchestra.run.smithi103.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-16T11:13:32.076 INFO:teuthology.orchestra.run.smithi103.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 390 kB/s | 16 kB 00:00 2024-09-16T11:13:32.102 INFO:teuthology.orchestra.run.smithi103.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T11:13:32.127 INFO:teuthology.orchestra.run.smithi103.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-16T11:13:32.152 INFO:teuthology.orchestra.run.smithi103.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-16T11:13:32.195 INFO:teuthology.orchestra.run.smithi103.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-16T11:13:32.204 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:32.220 INFO:teuthology.orchestra.run.smithi103.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 298 kB/s | 20 kB 00:00 2024-09-16T11:13:32.245 INFO:teuthology.orchestra.run.smithi103.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-16T11:13:32.249 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:32.249 INFO:teuthology.orchestra.run.smithi103.stdout:Total 14 MB/s | 27 MB 00:02 2024-09-16T11:13:32.381 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:13:32.419 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:13:32.419 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:13:32.447 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:32.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:32.447 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:32.447 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:32.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:32.448 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:32.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T11:13:32.821 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:13:32.822 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:13:33.255 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:33.393 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:33.394 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:33.395 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:33.395 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:13:33.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:33.395 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 137 k 2024-09-16T11:13:33.395 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 379 k 2024-09-16T11:13:33.395 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:33.640 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 561 kB/s | 137 kB 00:00 2024-09-16T11:13:33.641 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:33.641 INFO:teuthology.orchestra.run.smithi001.stdout:Total 557 kB/s | 137 kB 00:00 2024-09-16T11:13:33.641 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:33.649 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:33.649 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:33.709 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:33.709 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:33.861 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:33.933 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:13:33.959 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:13:33.959 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:33.959 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T11:13:33.959 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:34.062 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:13:34.124 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T11:13:34.155 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T11:13:34.197 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T11:13:34.288 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T11:13:34.375 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T11:13:34.416 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T11:13:34.463 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T11:13:34.493 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T11:13:34.528 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T11:13:34.561 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T11:13:34.602 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T11:13:34.636 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T11:13:34.671 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T11:13:34.719 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T11:13:34.760 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:13:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T11:13:34.856 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:13:34.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:34.857 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:34.857 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:34.857 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:34.857 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:34.973 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T11:13:35.051 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T11:13:35.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-09-16T11:13:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T11:13:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T11:13:35.209 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T11:13:35.265 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T11:13:35.461 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T11:13:35.507 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T11:13:35.585 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T11:13:35.647 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:35.681 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T11:13:35.722 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T11:13:35.740 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:13:35.785 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:35.786 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:13:35.786 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T11:13:35.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T11:13:35.986 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T11:13:36.023 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:13:36.050 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:13:36.050 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:36.050 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T11:13:36.050 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:13:36.050 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:13:36.050 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:36.443 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:36.583 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:36.584 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:36.584 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:13:36.584 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:36.584 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:36.584 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T11:13:36.584 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout:Install 7 Packages 2024-09-16T11:13:36.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:36.586 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.7 M 2024-09-16T11:13:36.586 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 80 M 2024-09-16T11:13:36.586 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:36.810 INFO:teuthology.orchestra.run.smithi001.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 127 kB/s | 24 kB 00:00 2024-09-16T11:13:36.827 INFO:teuthology.orchestra.run.smithi001.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 70 kB/s | 15 kB 00:00 2024-09-16T11:13:36.860 INFO:teuthology.orchestra.run.smithi001.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-09-16T11:13:36.886 INFO:teuthology.orchestra.run.smithi001.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-16T11:13:36.911 INFO:teuthology.orchestra.run.smithi001.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 488 kB/s | 41 kB 00:00 2024-09-16T11:13:36.945 INFO:teuthology.orchestra.run.smithi001.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-16T11:13:37.358 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:13:37.385 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:13:37.386 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:37.386 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T11:13:37.386 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:13:37.386 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:13:37.386 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:37.767 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:13:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:13:37.791 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:37.792 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T11:13:37.792 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:13:37.792 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:13:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:37.820 INFO:teuthology.orchestra.run.smithi001.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 2.8 MB/s | 3.4 MB 00:01 2024-09-16T11:13:37.824 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:37.825 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.9 MB/s | 3.7 MB 00:01 2024-09-16T11:13:37.825 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:37.835 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:37.835 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:37.933 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:37.933 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:37.993 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:13:38.017 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:13:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T11:13:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:13:38.017 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:13:38.017 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:38.081 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:13:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:38.321 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T11:13:38.356 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T11:13:38.413 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T11:13:38.442 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T11:13:38.471 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T11:13:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T11:13:39.583 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:13:39.721 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:13:39.721 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T11:13:39.721 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T11:13:39.722 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T11:13:39.723 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T11:13:39.724 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T11:13:39.725 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T11:13:40.517 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:13:40.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T11:13:40.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T11:13:40.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T11:13:40.517 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T11:13:40.518 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T11:13:40.518 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T11:13:40.614 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T11:13:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T11:13:40.616 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T11:13:40.617 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:40.618 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:40.815 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T11:13:40.816 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T11:13:40.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:40.817 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:40.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-09-16T11:13:41.000 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T11:13:41.554 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:41.590 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:41.645 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:13:41.687 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:41.688 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:13:41.688 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:41.729 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T11:13:41.730 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-09-16T11:13:41.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 38 M 2024-09-16T11:13:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 212 M 2024-09-16T11:13:41.732 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:41.740 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-09-16T11:13:42.121 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-16T11:13:42.188 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-16T11:13:42.306 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:42.439 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 83 k 2024-09-16T11:13:42.440 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 375 k 2024-09-16T11:13:42.441 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:13:42.563 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 13 MB/s | 7.4 MB 00:00 2024-09-16T11:13:42.661 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 376 kB/s | 83 kB 00:00 2024-09-16T11:13:42.661 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:42.662 INFO:teuthology.orchestra.run.smithi103.stdout:Total 373 kB/s | 83 kB 00:00 2024-09-16T11:13:42.662 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:13:42.667 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:13:42.667 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:13:42.669 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:13:42.669 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:13:42.742 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:13:42.755 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-09-16T11:13:43.047 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-16T11:13:43.202 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:43.298 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:43.331 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-16T11:13:43.364 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 63 kB/s | 38 kB 00:00 2024-09-16T11:13:43.636 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:13:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:43.886 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:13:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:43.886 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:44.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T11:13:44.148 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 303 kB/s | 245 kB 00:00 2024-09-16T11:13:44.331 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 3.9 MB/s | 4.9 MB 00:01 2024-09-16T11:13:44.498 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-16T11:13:44.638 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:09 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:44.771 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:44.772 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:44.772 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T11:13:44.772 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:44.772 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 137 k 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 379 k 2024-09-16T11:13:44.773 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:13:45.006 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 292 kB/s | 250 kB 00:00 2024-09-16T11:13:45.235 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 298 kB/s | 137 kB 00:00 2024-09-16T11:13:45.236 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:45.236 INFO:teuthology.orchestra.run.smithi103.stdout:Total 296 kB/s | 137 kB 00:00 2024-09-16T11:13:45.236 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:13:45.243 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:13:45.244 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:13:45.301 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:13:45.301 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:13:45.447 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:13:45.528 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:13:45.553 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:13:45.553 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:13:45.553 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T11:13:45.553 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:45.765 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T11:13:45.766 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:45.767 INFO:teuthology.orchestra.run.smithi001.stdout:Total 9.5 MB/s | 38 MB 00:04 2024-09-16T11:13:46.139 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:46.168 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:46.168 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:46.439 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:13:46.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:13:46.440 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:13:46.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:46.440 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:46.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-09-16T11:13:46.693 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:46.693 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:47.230 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:47.325 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:13:47.369 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:13:47.371 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:47.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T11:13:47.487 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:47.548 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T11:13:47.596 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:13:47.704 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:13:47.748 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T11:13:47.933 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T11:13:47.972 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T11:13:48.029 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:13 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:48.052 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T11:13:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T11:13:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:Install 7 Packages 2024-09-16T11:13:48.165 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.7 M 2024-09-16T11:13:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 80 M 2024-09-16T11:13:48.166 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:13:48.516 INFO:teuthology.orchestra.run.smithi103.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 161 kB/s | 24 kB 00:00 2024-09-16T11:13:48.541 INFO:teuthology.orchestra.run.smithi103.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 83 kB/s | 15 kB 00:00 2024-09-16T11:13:48.583 INFO:teuthology.orchestra.run.smithi103.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-16T11:13:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-16T11:13:48.634 INFO:teuthology.orchestra.run.smithi103.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-16T11:13:48.659 INFO:teuthology.orchestra.run.smithi103.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T11:13:48.760 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T11:13:48.801 INFO:teuthology.orchestra.run.smithi103.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 7.7 MB/s | 3.4 MB 00:00 2024-09-16T11:13:48.805 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:48.805 INFO:teuthology.orchestra.run.smithi103.stdout:Total 5.7 MB/s | 3.7 MB 00:00 2024-09-16T11:13:48.805 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:13:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:13:48.813 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:13:48.882 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T11:13:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:13:48.896 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:13:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:13:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T11:13:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T11:13:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T11:13:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T11:13:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T11:13:50.206 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:13:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T11:13:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T11:13:51.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T11:13:51.473 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T11:13:51.512 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:13:51.718 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T11:13:51.718 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:51.718 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:13:51.718 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:13:51.920 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T11:13:52.488 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:17 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:52.624 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:13:52.625 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-09-16T11:13:52.625 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:13:52.625 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-09-16T11:13:52.625 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T11:13:52.626 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 38 M 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 212 M 2024-09-16T11:13:52.627 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:13:52.943 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-16T11:13:53.043 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-16T11:13:53.069 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-16T11:13:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-09-16T11:13:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 462 kB/s | 38 kB 00:00 2024-09-16T11:13:53.445 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 11 MB/s | 7.4 MB 00:00 2024-09-16T11:13:53.587 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2024-09-16T11:13:53.771 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 752 kB/s | 245 kB 00:00 2024-09-16T11:13:53.887 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T11:13:53.995 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T11:13:53.996 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:13:53.996 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T11:13:54.381 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:54.382 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:13:54.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-09-16T11:13:54.879 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.9 MB/s | 4.9 MB 00:01 2024-09-16T11:13:55.380 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:13:55.521 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T11:13:55.522 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:13:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:13:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-09-16T11:13:55.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:13:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.2 M 2024-09-16T11:13:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-09-16T11:13:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:13:55.888 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 264 kB/s | 47 kB 00:00 2024-09-16T11:13:56.079 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.0 MB/s | 5.1 MB 00:02 2024-09-16T11:13:56.263 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 34 kB/s | 19 kB 00:00 2024-09-16T11:13:56.288 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 44 kB/s | 26 kB 00:00 2024-09-16T11:13:56.714 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 269 kB/s | 222 kB 00:00 2024-09-16T11:13:56.981 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 222 kB/s | 159 kB 00:00 2024-09-16T11:13:57.031 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 651 kB/s | 32 kB 00:00 2024-09-16T11:13:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 583 kB/s | 14 kB 00:00 2024-09-16T11:13:57.089 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-16T11:13:57.123 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 334 kB/s | 279 kB 00:00 2024-09-16T11:13:57.173 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 12 MB/s | 1.0 MB 00:00 2024-09-16T11:13:57.190 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 889 kB/s | 59 kB 00:00 2024-09-16T11:13:57.216 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-09-16T11:13:57.357 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 83 kB/s | 54 kB 00:00 2024-09-16T11:13:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:13:57.361 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-16T11:13:57.435 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:13:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:13:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:13:57.549 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:13:57.549 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:13:58.073 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:13:58.239 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T11:13:58.295 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T11:13:58.359 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T11:13:58.402 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T11:13:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T11:13:58.559 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T11:13:58.669 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T11:13:58.721 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T11:13:59.484 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T11:13:59.534 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T11:13:59.567 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T11:13:59.614 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T11:13:59.637 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:14:00.396 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:14:00.396 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T11:14:00.396 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T11:14:00.396 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T11:14:00.397 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T11:14:00.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T11:14:00.398 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T11:14:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T11:14:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T11:14:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:00.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T11:14:01.146 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-16T11:14:01.147 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:01.148 INFO:teuthology.orchestra.run.smithi103.stdout:Total 4.5 MB/s | 38 MB 00:08 2024-09-16T11:14:01.506 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:01.532 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:01.532 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:01.582 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:01.721 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:01.735 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:01.736 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:01.737 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:01.737 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-09-16T11:14:01.737 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:01.737 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 7.0 M 2024-09-16T11:14:01.737 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-09-16T11:14:01.737 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:02.043 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:02.176 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 568 kB/s | 124 kB 00:00 2024-09-16T11:14:02.242 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T11:14:02.268 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-16T11:14:02.293 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-16T11:14:02.760 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 309 kB/s | 248 kB 00:00 2024-09-16T11:14:02.802 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:02.876 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T11:14:02.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:14:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:14:03.062 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T11:14:03.143 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 6.0 MB 00:01 2024-09-16T11:14:03.145 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:03.146 INFO:teuthology.orchestra.run.smithi001.stdout:Total 4.9 MB/s | 7.0 MB 00:01 2024-09-16T11:14:03.222 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:03.235 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:03.235 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:03.254 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T11:14:03.286 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T11:14:03.341 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:03.342 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:03.371 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T11:14:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:03.842 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T11:14:03.896 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T11:14:04.028 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T11:14:04.141 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T11:14:04.456 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T11:14:04.554 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T11:14:04.630 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T11:14:04.657 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:14:05.393 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:14:05.393 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T11:14:05.393 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T11:14:05.393 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T11:14:05.393 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T11:14:05.393 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T11:14:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T11:14:05.662 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T11:14:05.663 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:05.663 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:05.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-09-16T11:14:06.423 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T11:14:06.445 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:14:06.494 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T11:14:06.634 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 893 k 2024-09-16T11:14:06.635 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.6 M 2024-09-16T11:14:06.636 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:06.812 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 514 kB/s | 80 kB 00:00 2024-09-16T11:14:06.970 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.5 MB/s | 813 kB 00:00 2024-09-16T11:14:06.971 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:06.971 INFO:teuthology.orchestra.run.smithi001.stdout:Total 2.6 MB/s | 893 kB 00:00 2024-09-16T11:14:06.983 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:07.005 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:07.005 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:07.089 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:07.089 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:07.289 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:07.435 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T11:14:07.506 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:14:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:14:08.179 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T11:14:08.443 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T11:14:08.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:08.444 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:08.444 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T11:14:08.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:08.444 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:08.669 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:14:08.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T11:14:08.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-volume 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T11:14:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:09.090 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:09.283 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:09.421 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:09.422 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-09-16T11:14:09.422 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:09.422 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T11:14:09.422 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 270 k 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 1.3 M 2024-09-16T11:14:09.423 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:09.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-09-16T11:14:09.666 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-16T11:14:09.667 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:09.667 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-16T11:14:09.667 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:09.676 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:09.677 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:09.708 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:09.708 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:09.927 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:09.985 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:14:10.001 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:14:10.001 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:14:10.002 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T11:14:10.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:10.024 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:10.156 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:10.157 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T11:14:10.158 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-09-16T11:14:10.159 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:10.160 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.2 M 2024-09-16T11:14:10.160 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-09-16T11:14:10.160 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:10.499 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 214 kB/s | 19 kB 00:00 2024-09-16T11:14:10.516 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 240 kB/s | 26 kB 00:00 2024-09-16T11:14:10.583 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-16T11:14:10.625 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T11:14:10.658 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-09-16T11:14:10.675 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 178 kB/s | 47 kB 00:00 2024-09-16T11:14:10.692 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 796 kB/s | 54 kB 00:00 2024-09-16T11:14:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 384 kB/s | 32 kB 00:00 2024-09-16T11:14:10.759 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-16T11:14:10.785 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T11:14:10.810 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-16T11:14:10.835 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T11:14:10.869 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-09-16T11:14:10.872 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:10.872 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-16T11:14:10.934 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:10.949 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:10.949 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:11.036 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:11.038 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:11.061 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:14:11.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:11.062 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:11.062 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:11.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:11.062 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:11.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-09-16T11:14:11.538 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:11.716 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T11:14:11.770 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T11:14:11.834 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T11:14:11.871 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T11:14:11.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T11:14:11.951 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:12.016 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T11:14:12.090 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 126 k 2024-09-16T11:14:12.091 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 448 k 2024-09-16T11:14:12.092 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:12.124 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T11:14:12.173 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T11:14:12.279 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 673 kB/s | 126 kB 00:00 2024-09-16T11:14:12.280 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:12.280 INFO:teuthology.orchestra.run.smithi001.stdout:Total 669 kB/s | 126 kB 00:00 2024-09-16T11:14:12.280 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:12.286 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:12.286 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:12.352 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:12.353 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:12.511 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:12.614 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T11:14:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T11:14:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T11:14:13.025 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T11:14:13.050 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:14:13.060 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:13.312 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:13.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:13.312 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:13.312 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:13.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:13.312 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:13.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T11:14:13.847 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T11:14:13.848 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T11:14:13.848 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T11:14:13.848 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T11:14:13.848 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T11:14:14.128 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T11:14:14.165 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T11:14:14.166 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T11:14:14.166 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T11:14:14.166 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T11:14:14.166 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T11:14:14.167 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T11:14:14.167 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T11:14:14.167 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:14.167 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:14.218 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:14.264 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:14.265 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:14.265 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:14.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-09-16T11:14:14.417 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T11:14:14.936 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:14.982 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:15.075 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:15.076 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:14:15.077 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:15.077 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 25 k 2024-09-16T11:14:15.077 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 110 k 2024-09-16T11:14:15.077 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:15.116 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:15.116 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:15.117 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:15.118 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:15.118 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-09-16T11:14:15.118 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:15.118 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 7.0 M 2024-09-16T11:14:15.118 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-09-16T11:14:15.118 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:15.241 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 152 kB/s | 25 kB 00:00 2024-09-16T11:14:15.241 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:15.242 INFO:teuthology.orchestra.run.smithi001.stdout:Total 151 kB/s | 25 kB 00:00 2024-09-16T11:14:15.242 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:15.245 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:15.245 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:15.247 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:15.247 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:15.335 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:15.364 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 643 kB/s | 124 kB 00:00 2024-09-16T11:14:15.440 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T11:14:15.440 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:15.465 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T11:14:15.490 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T11:14:15.757 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 424 kB/s | 248 kB 00:00 2024-09-16T11:14:15.829 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:16.076 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:16.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:16.076 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:16.076 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:16.076 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:16.076 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:16.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-09-16T11:14:16.844 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:16.938 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:16.984 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:16.985 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:16.985 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:17.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-09-16T11:14:17.655 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:17.724 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.3 MB/s | 6.0 MB 00:02 2024-09-16T11:14:17.726 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:17.727 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.7 MB/s | 7.0 MB 00:02 2024-09-16T11:14:17.747 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:17.793 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:17.794 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:17.794 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:17.801 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:17.813 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:17.813 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:17.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-09-16T11:14:17.913 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:17.914 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:18.381 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T11:14:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T11:14:18.447 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:18.541 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:18.586 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:18.587 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:18.587 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:18.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-09-16T11:14:18.949 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T11:14:19.028 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T11:14:19.084 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T11:14:19.113 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:14:19.251 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:19.346 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:19.392 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:19.393 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:19.393 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:19.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-09-16T11:14:19.836 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:14:19.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T11:14:19.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T11:14:19.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T11:14:19.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T11:14:19.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T11:14:20.055 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:20.114 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T11:14:20.114 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:20.114 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:20.115 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:20.149 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:20.194 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:20.195 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:20.195 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:20.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-09-16T11:14:20.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-09-16T11:14:20.857 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:20.926 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:20.951 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:20.996 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:20.997 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:20.997 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:21.059 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:21.060 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-09-16T11:14:21.061 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:21.061 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 893 k 2024-09-16T11:14:21.061 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.6 M 2024-09-16T11:14:21.061 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:21.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-09-16T11:14:21.336 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 544 kB/s | 80 kB 00:00 2024-09-16T11:14:21.586 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.0 MB/s | 813 kB 00:00 2024-09-16T11:14:21.586 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:21.587 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.7 MB/s | 893 kB 00:00 2024-09-16T11:14:21.598 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:21.620 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:21.620 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:21.680 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:21.701 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:21.701 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:21.824 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:14:21.825 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:21.826 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 82 k 2024-09-16T11:14:21.826 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 222 k 2024-09-16T11:14:21.826 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:21.986 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 511 kB/s | 82 kB 00:00 2024-09-16T11:14:21.987 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:21.987 INFO:teuthology.orchestra.run.smithi001.stdout:Total 505 kB/s | 82 kB 00:00 2024-09-16T11:14:21.987 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:21.995 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:21.995 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:22.036 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T11:14:22.064 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:22.064 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:14:22.212 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:22.319 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:22.710 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:22.796 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:14:22.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T11:14:22.985 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:22.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:22.985 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:22.985 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:22.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:22.985 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T11:14:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:23.071 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T11:14:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:23.072 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:23.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-09-16T11:14:23.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2024-09-16T11:14:23.828 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:23.927 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:23.968 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:23.969 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:23.970 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:14:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:23.970 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.9 M 2024-09-16T11:14:23.970 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 12 M 2024-09-16T11:14:23.970 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:24.061 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:24.061 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:24.061 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 270 k 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.3 M 2024-09-16T11:14:24.062 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:24.292 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-16T11:14:24.292 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:24.293 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-16T11:14:24.293 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:24.297 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 9.0 MB/s | 2.9 MB 00:00 2024-09-16T11:14:24.298 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:24.298 INFO:teuthology.orchestra.run.smithi001.stdout:Total 8.9 MB/s | 2.9 MB 00:00 2024-09-16T11:14:24.299 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:24.301 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:24.301 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:24.308 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:24.308 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:24.326 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:24.327 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:24.392 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:24.393 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:24.537 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:24.611 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:14:24.628 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:14:24.628 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:14:24.628 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T11:14:24.628 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:24.926 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:14:24.926 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T11:14:24.926 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:14:24.926 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:14:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:25.657 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:14:25.657 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:25.657 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:25.658 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:14:25.658 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:25.658 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:25.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-09-16T11:14:26.018 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:26.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:26.018 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:26.018 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:26.018 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:26.018 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:26.237 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-09-16T11:14:26.493 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:26.628 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:26.629 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:26.630 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:26.630 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:14:26.630 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:26.630 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 126 k 2024-09-16T11:14:26.630 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 448 k 2024-09-16T11:14:26.630 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:26.834 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:26.840 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 600 kB/s | 126 kB 00:00 2024-09-16T11:14:26.841 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:26.843 INFO:teuthology.orchestra.run.smithi103.stdout:Total 595 kB/s | 126 kB 00:00 2024-09-16T11:14:26.843 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:26.847 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:26.847 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:26.910 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:26.910 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:26.973 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:26.973 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:26.973 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-09-16T11:14:26.974 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:26.975 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 165 k 2024-09-16T11:14:26.975 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 477 k 2024-09-16T11:14:26.975 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-09-16T11:14:27.060 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:27.171 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:27.192 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 760 kB/s | 165 kB 00:00 2024-09-16T11:14:27.192 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:27.193 INFO:teuthology.orchestra.run.smithi001.stdout:Total 755 kB/s | 165 kB 00:00 2024-09-16T11:14:27.193 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-09-16T11:14:27.201 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-09-16T11:14:27.201 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-09-16T11:14:27.268 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-09-16T11:14:27.269 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-09-16T11:14:27.430 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-09-16T11:14:27.543 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:27.589 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:27.838 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:27.838 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:27.838 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:27.839 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:27.839 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:27.839 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:27.992 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:28.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-09-16T11:14:28.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:28.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:28.207 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-09-16T11:14:28.207 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:28.208 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:28.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-09-16T11:14:28.643 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:28.734 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:28.779 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:28.780 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:28.780 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:28.833 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-09-16T11:14:29.024 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:54 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:29.118 INFO:teuthology.orchestra.run.smithi001.stdout:Package rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:29.164 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-09-16T11:14:29.165 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-09-16T11:14:29.165 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-09-16T11:14:29.236 DEBUG:teuthology.parallel:result is None 2024-09-16T11:14:29.403 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:54 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:29.536 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:29.536 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:29.536 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T11:14:29.536 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:29.536 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:29.537 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 25 k 2024-09-16T11:14:29.538 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 110 k 2024-09-16T11:14:29.538 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:29.687 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 167 kB/s | 25 kB 00:00 2024-09-16T11:14:29.687 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:29.688 INFO:teuthology.orchestra.run.smithi103.stdout:Total 165 kB/s | 25 kB 00:00 2024-09-16T11:14:29.688 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:29.691 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:29.692 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:29.693 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:29.696 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:29.789 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:29.893 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:30.263 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:30.495 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:30.495 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:30.496 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:30.496 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:30.496 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:30.496 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:30.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-09-16T11:14:31.227 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:31.319 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:31.361 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:31.363 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:31.363 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:31.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-09-16T11:14:31.986 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:32.076 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:32.119 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:32.120 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:32.121 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:32.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-09-16T11:14:32.736 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:57 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:32.825 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:32.867 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:32.868 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:32.869 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:32.930 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-09-16T11:14:33.489 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:58 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:33.578 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:33.621 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:33.622 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:33.622 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:33.675 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-09-16T11:14:34.234 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:59 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:34.323 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:34.366 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:34.367 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:34.367 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:34.423 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-09-16T11:14:34.981 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:59 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:35.071 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:35.114 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:35.115 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:35.115 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:35.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-09-16T11:14:35.744 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:00 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:35.877 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:35.877 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 82 k 2024-09-16T11:14:35.878 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 222 k 2024-09-16T11:14:35.879 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:36.074 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 418 kB/s | 82 kB 00:00 2024-09-16T11:14:36.075 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:36.075 INFO:teuthology.orchestra.run.smithi103.stdout:Total 416 kB/s | 82 kB 00:00 2024-09-16T11:14:36.075 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:36.082 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:36.082 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:36.146 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:36.146 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:36.287 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:36.379 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:36.743 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:37.015 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:37.015 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:37.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-09-16T11:14:37.815 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:02 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:37.953 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:37.954 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.9 M 2024-09-16T11:14:37.955 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 12 M 2024-09-16T11:14:37.955 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:38.304 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.4 MB/s | 2.9 MB 00:00 2024-09-16T11:14:38.304 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:38.305 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-09-16T11:14:38.305 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:38.314 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:38.314 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:38.398 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:38.398 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:38.816 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:38.888 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:38.912 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:38.913 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:14:38.913 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T11:14:38.913 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:14:38.913 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:14:38.913 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:40.003 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:40.003 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:40.003 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:40.003 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:40.003 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:40.003 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:40.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-09-16T11:14:40.812 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:05 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:40.949 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-09-16T11:14:40.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:40.951 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 165 k 2024-09-16T11:14:40.951 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 477 k 2024-09-16T11:14:40.951 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-16T11:14:41.425 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 348 kB/s | 165 kB 00:00 2024-09-16T11:14:41.425 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:14:41.426 INFO:teuthology.orchestra.run.smithi103.stdout:Total 347 kB/s | 165 kB 00:00 2024-09-16T11:14:41.426 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-16T11:14:41.434 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-16T11:14:41.434 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-16T11:14:41.500 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-16T11:14:41.500 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-16T11:14:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-16T11:14:41.749 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:42.182 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:14:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:42.451 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-16T11:14:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:14:42.451 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:42.451 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:42.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-09-16T11:14:43.268 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:08 ago on Mon 16 Sep 2024 11:12:35 AM UTC. 2024-09-16T11:14:43.361 INFO:teuthology.orchestra.run.smithi103.stdout:Package rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:14:43.403 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-16T11:14:43.405 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-09-16T11:14:43.405 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-16T11:14:43.465 DEBUG:teuthology.parallel:result is None 2024-09-16T11:14:43.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:14:43.607 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T11:14:43.634 INFO:teuthology.orchestra.run.smithi001.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T11:14:43.634 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T11:14:43.634 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T11:14:43.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:14:43.791 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T11:14:43.812 INFO:teuthology.orchestra.run.smithi103.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T11:14:43.812 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T11:14:43.812 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T11:14:43.814 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T11:14:43.815 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:43.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T11:14:43.845 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:43.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T11:14:43.878 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T11:14:43.878 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:43.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T11:14:43.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T11:14:43.987 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:43.987 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T11:14:44.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T11:14:44.077 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T11:14:44.077 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:44.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T11:14:44.108 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T11:14:44.177 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:44.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T11:14:44.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T11:14:44.266 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T11:14:44.266 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:44.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T11:14:44.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T11:14:44.365 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:44.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T11:14:44.390 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T11:14:44.455 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T11:14:44.538 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T11:14:44.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-09-16T11:14:44.541 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-09-16T11:14:44.569 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T11:14:44.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T11:14:44.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T11:14:44.640 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T11:14:44.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T11:14:44.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T11:14:44.727 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-16T11:14:44.728 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T11:14:44.728 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907167', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGqAFr/poL95sO4gd9TCrLiYuCVqFzL5efo5srdCOpECOtvr6+2wExCxiQcB5UD5JOjszyCNZqaCh2dYXBmp8Y=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIn6j46Ue+ntpffguTLbpcQNYJduNZD9odWBx7pmD0KsupCXWd35E+7vLdXpX1BFhsiUSaYof4nIDQ6cVd+aL2M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T11:14:44.728 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T11:14:44.752 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T11:14:44.768 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:14:44.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:14:44.836 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:44.836 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T11:14:44.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T11:14:44.897 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-09-16T11:14:44.955 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T11:14:44.956 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:14:44.956 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T11:14:44.956 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:14:44.956 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:14:44.956 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-16 11:14:25.181187402 +0000 2024-09-16T11:14:44.957 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-16 11:11:56.859475673 +0000 2024-09-16T11:14:44.957 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-16 11:11:56.859475673 +0000 2024-09-16T11:14:44.957 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-16 11:11:56.859475673 +0000 2024-09-16T11:14:44.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T11:14:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-16T11:14:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-16T11:14:45.023 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000107381 s, 4.8 MB/s 2024-09-16T11:14:45.024 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T11:14:45.083 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-09-16T11:14:45.140 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T11:14:45.140 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:14:45.140 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T11:14:45.140 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:14:45.141 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:14:45.141 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-16 11:14:25.181187402 +0000 2024-09-16T11:14:45.141 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-16 11:11:57.191471496 +0000 2024-09-16T11:14:45.141 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-16 11:11:57.191471496 +0000 2024-09-16T11:14:45.141 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-16 11:11:57.191471496 +0000 2024-09-16T11:14:45.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T11:14:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-16T11:14:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-16T11:14:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000110818 s, 4.6 MB/s 2024-09-16T11:14:45.209 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T11:14:45.267 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-09-16T11:14:45.325 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T11:14:45.325 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-16 11:14:25.182187390 +0000 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-16 11:11:57.510467481 +0000 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-16 11:11:57.510467481 +0000 2024-09-16T11:14:45.326 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-16 11:11:57.510467481 +0000 2024-09-16T11:14:45.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T11:14:45.393 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-16T11:14:45.393 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-16T11:14:45.393 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000113724 s, 4.5 MB/s 2024-09-16T11:14:45.394 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T11:14:45.452 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-09-16T11:14:45.513 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T11:14:45.513 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-09-16 11:14:25.182187390 +0000 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-09-16 11:11:57.825463518 +0000 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-09-16 11:11:57.825463518 +0000 2024-09-16T11:14:45.514 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-09-16 11:11:57.825463518 +0000 2024-09-16T11:14:45.514 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T11:14:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-09-16T11:14:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-09-16T11:14:45.580 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.3392e-05 s, 5.5 MB/s 2024-09-16T11:14:45.581 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T11:14:45.638 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:14:45.638 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:14:45.639 INFO:tasks.ceph:Generating config... 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:45.640 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:45.641 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:45.641 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:45.641 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:45.641 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T11:14:45.641 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T11:14:45.641 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T11:14:45.641 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T11:14:45.641 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T11:14:45.641 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T11:14:45.641 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T11:14:45.641 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T11:14:45.641 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T11:14:45.642 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T11:14:45.643 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-16T11:14:45.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-16T11:14:45.726 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/cluster1.keyring 2024-09-16T11:14:45.728 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-16T11:14:45.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T11:14:45.836 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.1')] 2024-09-16T11:14:45.836 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T11:14:45.837 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:45.837 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:14:45.892 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid 36663229-3ca4-47f4-b225-543cee2c8a74 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = octopus 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 36663229-3ca4-47f4-b225-543cee2c8a74 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-09-16T11:14:45.968017+0000 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-09-16T11:14:45.968017+0000 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 15 (octopus) 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-09-16T11:14:45.969 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-16T11:14:45.972 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:14:46.029 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 36663229-3ca4-47f4-b225-543cee2c8a74... 2024-09-16T11:14:46.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T11:14:46.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-09-16T11:14:46.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.115 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-09-16T11:14:46.116 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-09-16T11:14:46.117 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = 172.21.15.1 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on pool no app = False 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 5000 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 36663229-3ca4-47f4-b225-543cee2c8a74 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.118 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-09-16T11:14:46.119 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-09-16T11:14:46.120 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-09-16T11:14:46.121 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-09-16T11:14:46.121 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-09-16T11:14:46.121 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = bitmap 2024-09-16T11:14:46.121 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-09-16T11:14:46.121 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-09-16T11:14:46.121 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-09-16T11:14:46.122 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-09-16T11:14:46.123 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.124 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:14:46.125 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd = 20 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd_mirror = 15 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: log to stderr = False 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: rbd default features = 125 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: rbd mirror image policy type = simple 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.0] 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.1] 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.126 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.2] 2024-09-16T11:14:46.127 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.127 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.127 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.3] 2024-09-16T11:14:46.127 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.127 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.127 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-09-16T11:14:46.128 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.129 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:14:46.130 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-09-16T11:14:46.131 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = 172.21.15.1 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: mon client directed command retry = 5 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on pool no app = False 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: ms inject socket failures = 5000 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 36663229-3ca4-47f4-b225-543cee2c8a74 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:14:46.132 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-09-16T11:14:46.133 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: bdev async discard = True 2024-09-16T11:14:46.134 INFO:teuthology.orchestra.run.smithi103.stdout: bdev enable discard = True 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore allocator = bitmap 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 1/20 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 1/20 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 4/10 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-09-16T11:14:46.135 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:14:46.136 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:14:46.137 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:14:46.138 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: debug rbd = 20 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: debug rbd_mirror = 15 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: log to stderr = False 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: rbd default features = 125 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: rbd mirror image policy type = simple 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.0] 2024-09-16T11:14:46.139 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.1] 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.2] 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.3] 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:14:46.140 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:14:46.142 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-16T11:14:46.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-16T11:14:46.234 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T11:14:46.235 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:46.235 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-16T11:14:46.252 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:46.252 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-16T11:14:46.308 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:14:46.308 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:46.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T11:14:46.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T11:14:46.384 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:46.384 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:46.439 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:14:46.440 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:46.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T11:14:46.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T11:14:46.481 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:46.481 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:46.539 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T11:14:46.540 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T11:14:46.540 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T11:14:46.588 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T11:14:46.591 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T11:14:46.591 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T11:14:46.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-16T11:14:46.630 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-16T11:14:46.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T11:14:46.693 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T11:14:46.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T11:14:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T11:14:46.760 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T11:14:46.839 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T11:14:46.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T11:14:46.930 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T11:14:46.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T11:14:47.020 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T11:14:47.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T11:14:47.109 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T11:14:47.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T11:14:47.198 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T11:14:47.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T11:14:47.288 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T11:14:47.300 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T11:14:47.300 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:14:47.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-16T11:14:47.332 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:14:47.332 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-16T11:14:47.332 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:14:47.332 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T11:14:47.401 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:14:47.401 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:14:47.401 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:14:47.401 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:14:47.401 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:14:47.402 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:14:47.402 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:14:47.402 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:14:47.402 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:14:47.402 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:14:47.434 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-16T11:14:47.436 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-16T11:14:47.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-16T11:14:47.508 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-16T11:14:47.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-16T11:14:47.646 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:14:47.647 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-16T11:14:47.647 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:14:47.647 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:14:47.712 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:14:47.713 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:14:47.713 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:14:47.713 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:14:47.748 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-16T11:14:47.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-16T11:14:47.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-16T11:14:47.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-16T11:14:47.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-16T11:14:47.981 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:14:47.981 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-16T11:14:47.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:14:47.982 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T11:14:48.047 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:14:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:14:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:14:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:14:48.047 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:14:48.047 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:14:48.048 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:14:48.048 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:14:48.048 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:14:48.048 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:14:48.080 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-09-16T11:14:48.082 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-09-16T11:14:48.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-16T11:14:48.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-16T11:14:48.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:48.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.327+0000 7f4ce660f440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-16T11:14:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.327+0000 7f4ce660f440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-16T11:14:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.327+0000 7f4ce660f440 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T11:14:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.327+0000 7f4ce660f440 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T11:14:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.327+0000 7f4ce660f440 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T11:14:48.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.327+0000 7f4ce660f440 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-16T11:14:48.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:48.860+0000 7f4ce660f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:49.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:49.627+0000 7f4ce660f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:50.391+0000 7f4ce660f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:51.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-16T11:14:51.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.312+0000 7f1092129440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-16T11:14:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.312+0000 7f1092129440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-16T11:14:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.312+0000 7f1092129440 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T11:14:51.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.312+0000 7f1092129440 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T11:14:51.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.312+0000 7f1092129440 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T11:14:51.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.313+0000 7f1092129440 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-16T11:14:51.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:51.841+0000 7f1092129440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:52.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:52.603+0000 7f1092129440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:53.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:53.364+0000 7f1092129440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:54.163 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-16T11:14:54.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.281+0000 7f4e2b233440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-16T11:14:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.281+0000 7f4e2b233440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-16T11:14:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.281+0000 7f4e2b233440 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T11:14:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.282+0000 7f4e2b233440 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T11:14:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.282+0000 7f4e2b233440 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T11:14:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.282+0000 7f4e2b233440 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-16T11:14:54.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:54.806+0000 7f4e2b233440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:55.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:55.572+0000 7f4e2b233440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:56.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-09-16T11:14:56.340+0000 7f4e2b233440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:14:57.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-16T11:14:57.166 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T11:14:57.166 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:57.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-16T11:14:57.236 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:57.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-16T11:14:57.305 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:57.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-16T11:14:57.373 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:57.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-16T11:14:57.440 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:57.440 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-16T11:14:57.497 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.497 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-16T11:14:57.517 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.517 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-16T11:14:57.576 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.576 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-16T11:14:57.635 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.635 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-16T11:14:57.693 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.693 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-16T11:14:57.750 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.751 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-16T11:14:57.808 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.808 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-16T11:14:57.864 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:14:57.864 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-16T11:14:57.921 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T11:14:57.924 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-16T11:14:57.953 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmE+hme2UMIxAAQ7zZAssFa3PToabl18XG8Q== 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAoE+hmZj+VExAA9qichZWuDufWL8jUlVAqYw== 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQArE+hmnY60EhAA/yEiiFl6MBaGcL0fheVBnA== 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAuE+hmPM3cEBAAXJBtgwmohzgjv+Vbu7kVGw== 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmE+hm+OiGJRAAvGkNTH0Slc+dbJ8FPacvNg== 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror] 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmE+hm+cdIKRAALFqqUmbn0QEyI4vBMNZs4g== 2024-09-16T11:14:57.954 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.0] 2024-09-16T11:14:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmE+hm0NyALBAA5ru+AnNBYd5Z7KCuXpYrlw== 2024-09-16T11:14:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.1] 2024-09-16T11:14:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmE+hmwRL8MRAANFnFxEJqZtPSwZAr1d8+NQ== 2024-09-16T11:14:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.2] 2024-09-16T11:14:57.955 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAmE+hmdxhnNxAAbI6ZdmG6Jkl9sJLpCQPcaw== 2024-09-16T11:14:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.3] 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAnE+hmkUElARAA5QbvgcqLKlR/S7FxO82a0Q== 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.4] 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAnE+hmWxF/BhAANydS/EvurY5Aew/PXSne2A== 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.5] 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAnE+hmFLfMCxAAetKMJdkmV6ErJdApxGuGTQ== 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.6] 2024-09-16T11:14:57.956 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQAnE+hmLfskERAAXHQZY20zQOYhTaP912g6KQ== 2024-09-16T11:14:57.957 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T11:14:58.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:14:58.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:14:58.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:14:58.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:14:58.668 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T11:14:58.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-16T11:14:58.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-16T11:14:58.789 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-16T11:14:58.815 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:14:58.872 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-16T11:14:58.872 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-16T11:14:58.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-16T11:14:58.914 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-16T11:14:58.914 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-16T11:14:58.914 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-16T11:14:58.915 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-16T11:14:58.917 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-16T11:14:58.917 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T11:14:58.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-16T11:14:59.087 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-09-16T11:14:59.098 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-16T11:14:59.098 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:59.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-16T11:14:59.124 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:59.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-16T11:14:59.191 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:14:59.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-16T11:14:59.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd new 8fd11c13-6bd9-4aea-9ff4-340755cd147b 0 2024-09-16T11:14:59.297 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:14:59.295+0000 7fa77f616ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T11:14:59.338 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:14:59.336+0000 7fa77f616ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T11:14:59.391 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:14:59.389+0000 7fa77f616ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T11:14:59.471 INFO:teuthology.orchestra.run.smithi001.stdout:0 2024-09-16T11:14:59.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd new 248e1432-8f0f-4e47-9dae-7b8d65418695 1 2024-09-16T11:14:59.530 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:14:59.528+0000 7fa77f616ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:14:59.645 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:14:59.643+0000 7fa77f616ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T11:14:59.696 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-09-16T11:14:59.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd new f66df91e-afd9-425e-bd74-2d5eb60bd0b7 2 2024-09-16T11:14:59.920 INFO:teuthology.orchestra.run.smithi001.stdout:2 2024-09-16T11:14:59.931 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-16T11:14:59.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-16T11:14:59.934 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-16T11:14:59.934 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-16T11:14:59.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-16T11:14:59.936 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-16T11:14:59.936 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-16T11:14:59.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-16T11:14:59.938 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-16T11:14:59.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:14:59.955 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:14:59.953+0000 7fa77f616ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T11:15:00.037 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:00.035+0000 7fa77f616ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T11:15:00.111 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:00.111 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:14:59.918514+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:00.122 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-16T11:15:00.122 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T11:15:00.177 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:00.175+0000 7fa77f616ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T11:15:00.425 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:15:00.542 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:15:00.540+0000 7f9851d21440 -1 Falling back to public interface 2024-09-16T11:15:00.543 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:15:00.541+0000 7f0011f21440 -1 Falling back to public interface 2024-09-16T11:15:00.570 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:15:00.568+0000 7f1fb260c440 -1 Falling back to public interface 2024-09-16T11:15:00.599 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-16T11:15:00.600 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":5,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:14:59.918514+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:00.609 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T11:15:01.145 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.143+0000 7fa77f616ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T11:15:01.197 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.195+0000 7fa77f616ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T11:15:01.578 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.576+0000 7fa77f616ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T11:15:01.663 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.661+0000 7fa77f616ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T11:15:01.756 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.754+0000 7fa77f616ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:15:01.802 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.800+0000 7fa77f616ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T11:15:01.901 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.899+0000 7fa77f616ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T11:15:01.955 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.953+0000 7fa77f616ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T11:15:02.000 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:01.998+0000 7fa77f616ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T11:15:02.097 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:02.095+0000 7fa77f616ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T11:15:02.148 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:02.146+0000 7fa77f616ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T11:15:02.201 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:02.199+0000 7fa77f616ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T11:15:02.270 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:02.268+0000 7fa77f616ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T11:15:02.314 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:02.312+0000 7fa77f616ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T11:15:02.359 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:02.357+0000 7fa77f616ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T11:15:03.002 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:03.000+0000 7fa77f616ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T11:15:04.045 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:15:04.043+0000 7f0011f21440 -1 osd.1 0 log_to_monitors true 2024-09-16T11:15:04.054 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:15:04.053+0000 7f9851d21440 -1 osd.0 0 log_to_monitors true 2024-09-16T11:15:04.071 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:15:04.069+0000 7f1fb260c440 -1 osd.2 0 log_to_monitors true 2024-09-16T11:15:04.210 INFO:tasks.ceph.cluster1.mgr.x.smithi001.stderr:2024-09-16T11:15:04.208+0000 7fa76417c640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T11:15:06.913 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:15:07.218 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:15:07.217+0000 7f000e1f9640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T11:15:07.223 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:15:07.221+0000 7f0005884640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:07.277 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:15:07.275+0000 7f1facdf6640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T11:15:07.281 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:15:07.279+0000 7f1fa5de8640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:07.306 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:15:07.304+0000 7f984c7f6640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T11:15:07.310 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:15:07.308+0000 7f9845684640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:07.338 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-16T11:15:07.338 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":7,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:15:06.008744+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:07.348 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T11:15:13.652 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:15:13.927 INFO:teuthology.misc.health.smithi001.stdout: 2024-09-16T11:15:13.927 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":11,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:15:11.221053+0000","last_up_change":"2024-09-16T11:15:08.218847+0000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:09.051608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":32155},{"type":"v1","addr":"172.21.15.1:6801","nonce":32155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":32155},{"type":"v1","addr":"172.21.15.1:6803","nonce":32155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":32155},{"type":"v1","addr":"172.21.15.1:6807","nonce":32155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":32155},{"type":"v1","addr":"172.21.15.1:6805","nonce":32155}]},"public_addr":"172.21.15.1:6801/32155","cluster_addr":"172.21.15.1:6803/32155","heartbeat_back_addr":"172.21.15.1:6807/32155","heartbeat_front_addr":"172.21.15.1:6805/32155","state":["exists","up"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":32156},{"type":"v1","addr":"172.21.15.1:6809","nonce":32156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":32156},{"type":"v1","addr":"172.21.15.1:6811","nonce":32156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":32156},{"type":"v1","addr":"172.21.15.1:6815","nonce":32156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":32156},{"type":"v1","addr":"172.21.15.1:6813","nonce":32156}]},"public_addr":"172.21.15.1:6809/32156","cluster_addr":"172.21.15.1:6811/32156","heartbeat_back_addr":"172.21.15.1:6815/32156","heartbeat_front_addr":"172.21.15.1:6813/32156","state":["exists","up"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":32185},{"type":"v1","addr":"172.21.15.1:6817","nonce":32185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":32185},{"type":"v1","addr":"172.21.15.1:6819","nonce":32185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":32185},{"type":"v1","addr":"172.21.15.1:6823","nonce":32185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":32185},{"type":"v1","addr":"172.21.15.1:6821","nonce":32185}]},"public_addr":"172.21.15.1:6817/32185","cluster_addr":"172.21.15.1:6819/32185","heartbeat_back_addr":"172.21.15.1:6823/32185","heartbeat_front_addr":"172.21.15.1:6821/32185","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.073960+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.045396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.041694+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:13.938 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T11:15:13.938 INFO:tasks.ceph:Creating RBD pool 2024-09-16T11:15:13.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-16T11:15:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'rbd' created 2024-09-16T11:15:14.306 DEBUG:teuthology.orchestra.run.smithi001:> rbd --cluster cluster1 pool init rbd 2024-09-16T11:15:17.309 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-16T11:15:17.310 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-16T11:15:17.310 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-16T11:15:17.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-16T11:15:17.310 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T11:15:17.635 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:17.645 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":31819},{"type":"v1","addr":"172.21.15.1:6825","nonce":31819}]},"active_addr":"172.21.15.1:6825/31819","active_change":"2024-09-16T11:15:03.002659+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":815748253}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2807007639}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":789814807}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3060527358}]}]} 2024-09-16T11:15:17.647 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-16T11:15:17.648 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-16T11:15:17.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:15:17.918 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:17.918 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:15:17.294904+0000","last_up_change":"2024-09-16T11:15:08.218847+0000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:09.051608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:14.238771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":32155},{"type":"v1","addr":"172.21.15.1:6801","nonce":32155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":32155},{"type":"v1","addr":"172.21.15.1:6803","nonce":32155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":32155},{"type":"v1","addr":"172.21.15.1:6807","nonce":32155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":32155},{"type":"v1","addr":"172.21.15.1:6805","nonce":32155}]},"public_addr":"172.21.15.1:6801/32155","cluster_addr":"172.21.15.1:6803/32155","heartbeat_back_addr":"172.21.15.1:6807/32155","heartbeat_front_addr":"172.21.15.1:6805/32155","state":["exists","up"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":32156},{"type":"v1","addr":"172.21.15.1:6809","nonce":32156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":32156},{"type":"v1","addr":"172.21.15.1:6811","nonce":32156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":32156},{"type":"v1","addr":"172.21.15.1:6815","nonce":32156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":32156},{"type":"v1","addr":"172.21.15.1:6813","nonce":32156}]},"public_addr":"172.21.15.1:6809/32156","cluster_addr":"172.21.15.1:6811/32156","heartbeat_back_addr":"172.21.15.1:6815/32156","heartbeat_front_addr":"172.21.15.1:6813/32156","state":["exists","up"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":32185},{"type":"v1","addr":"172.21.15.1:6817","nonce":32185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":32185},{"type":"v1","addr":"172.21.15.1:6819","nonce":32185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":32185},{"type":"v1","addr":"172.21.15.1:6823","nonce":32185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":32185},{"type":"v1","addr":"172.21.15.1:6821","nonce":32185}]},"public_addr":"172.21.15.1:6817/32185","cluster_addr":"172.21.15.1:6819/32185","heartbeat_back_addr":"172.21.15.1:6823/32185","heartbeat_front_addr":"172.21.15.1:6821/32185","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.073960+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.045396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.041694+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:17.930 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-16T11:15:17.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:15:18.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:18.204 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:15:17.294904+0000","last_up_change":"2024-09-16T11:15:08.218847+0000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:09.051608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:14.238771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":32155},{"type":"v1","addr":"172.21.15.1:6801","nonce":32155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":32155},{"type":"v1","addr":"172.21.15.1:6803","nonce":32155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":32155},{"type":"v1","addr":"172.21.15.1:6807","nonce":32155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":32155},{"type":"v1","addr":"172.21.15.1:6805","nonce":32155}]},"public_addr":"172.21.15.1:6801/32155","cluster_addr":"172.21.15.1:6803/32155","heartbeat_back_addr":"172.21.15.1:6807/32155","heartbeat_front_addr":"172.21.15.1:6805/32155","state":["exists","up"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":32156},{"type":"v1","addr":"172.21.15.1:6809","nonce":32156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":32156},{"type":"v1","addr":"172.21.15.1:6811","nonce":32156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":32156},{"type":"v1","addr":"172.21.15.1:6815","nonce":32156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":32156},{"type":"v1","addr":"172.21.15.1:6813","nonce":32156}]},"public_addr":"172.21.15.1:6809/32156","cluster_addr":"172.21.15.1:6811/32156","heartbeat_back_addr":"172.21.15.1:6815/32156","heartbeat_front_addr":"172.21.15.1:6813/32156","state":["exists","up"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":32185},{"type":"v1","addr":"172.21.15.1:6817","nonce":32185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":32185},{"type":"v1","addr":"172.21.15.1:6819","nonce":32185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":32185},{"type":"v1","addr":"172.21.15.1:6823","nonce":32185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":32185},{"type":"v1","addr":"172.21.15.1:6821","nonce":32185}]},"public_addr":"172.21.15.1:6817/32185","cluster_addr":"172.21.15.1:6819/32185","heartbeat_back_addr":"172.21.15.1:6823/32185","heartbeat_front_addr":"172.21.15.1:6821/32185","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.073960+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.045396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.041694+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:18.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-16T11:15:18.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-16T11:15:18.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-16T11:15:18.351 INFO:teuthology.orchestra.run.smithi001.stdout:34359738371 2024-09-16T11:15:18.351 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T11:15:18.372 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-16T11:15:18.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T11:15:18.373 INFO:teuthology.orchestra.run.smithi001.stdout:34359738373 2024-09-16T11:15:18.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T11:15:18.640 INFO:teuthology.orchestra.run.smithi001.stdout:34359738370 2024-09-16T11:15:18.652 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2024-09-16T11:15:18.698 INFO:teuthology.orchestra.run.smithi001.stdout:34359738371 2024-09-16T11:15:18.704 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-16T11:15:18.710 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-09-16T11:15:18.716 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-09-16T11:15:19.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T11:15:19.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T11:15:19.717 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T11:15:19.950 INFO:teuthology.orchestra.run.smithi001.stdout:34359738371 2024-09-16T11:15:19.961 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2024-09-16T11:15:19.962 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:20.001 INFO:teuthology.orchestra.run.smithi001.stdout:34359738372 2024-09-16T11:15:20.015 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-09-16T11:15:20.015 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:20.037 INFO:teuthology.orchestra.run.smithi001.stdout:34359738373 2024-09-16T11:15:20.049 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-09-16T11:15:20.049 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:20.049 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T11:15:20.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:15:20.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:20.361 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:15:20.373 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T11:15:19.009016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":906096,"kb_used_data":1548,"kb_used_omap":4,"kb_used_meta":904507,"kb_avail":282209424,"statfs":{"total":289910292480,"available":288982450176,"internally_reserved":0,"allocated":1585152,"data_stored":1294538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":926215176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.719177"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304322+0000","last_change":"2024-09-16T11:15:17.304545+0000","last_active":"2024-09-16T11:15:17.304322+0000","last_peered":"2024-09-16T11:15:17.304322+0000","last_clean":"2024-09-16T11:15:17.304322+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:15:17.304322+0000","last_undegraded":"2024-09-16T11:15:17.304322+0000","last_fullsized":"2024-09-16T11:15:17.304322+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:52.627160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304116+0000","last_change":"2024-09-16T11:15:17.304329+0000","last_active":"2024-09-16T11:15:17.304116+0000","last_peered":"2024-09-16T11:15:17.304116+0000","last_clean":"2024-09-16T11:15:17.304116+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:15:17.304116+0000","last_undegraded":"2024-09-16T11:15:17.304116+0000","last_fullsized":"2024-09-16T11:15:17.304116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:28.342868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304002+0000","last_change":"2024-09-16T11:15:17.304273+0000","last_active":"2024-09-16T11:15:17.304002+0000","last_peered":"2024-09-16T11:15:17.304002+0000","last_clean":"2024-09-16T11:15:17.304002+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:15:17.304002+0000","last_undegraded":"2024-09-16T11:15:17.304002+0000","last_fullsized":"2024-09-16T11:15:17.304002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:52.193450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304628+0000","last_change":"2024-09-16T11:15:17.304918+0000","last_active":"2024-09-16T11:15:17.304628+0000","last_peered":"2024-09-16T11:15:17.304628+0000","last_clean":"2024-09-16T11:15:17.304628+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:15:17.304628+0000","last_undegraded":"2024-09-16T11:15:17.304628+0000","last_fullsized":"2024-09-16T11:15:17.304628+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:24:48.683341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.302378+0000","last_change":"2024-09-16T11:15:17.302378+0000","last_active":"2024-09-16T11:15:17.302378+0000","last_peered":"2024-09-16T11:15:17.302378+0000","last_clean":"2024-09-16T11:15:17.302378+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:15:17.302378+0000","last_undegraded":"2024-09-16T11:15:17.302378+0000","last_fullsized":"2024-09-16T11:15:17.302378+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.265636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.299500+0000","last_change":"2024-09-16T11:15:17.299727+0000","last_active":"2024-09-16T11:15:17.299500+0000","last_peered":"2024-09-16T11:15:17.299500+0000","last_clean":"2024-09-16T11:15:17.299500+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:15:17.299500+0000","last_undegraded":"2024-09-16T11:15:17.299500+0000","last_fullsized":"2024-09-16T11:15:17.299500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:10.958157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.299412+0000","last_change":"2024-09-16T11:15:17.299646+0000","last_active":"2024-09-16T11:15:17.299412+0000","last_peered":"2024-09-16T11:15:17.299412+0000","last_clean":"2024-09-16T11:15:17.299412+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:15:17.299412+0000","last_undegraded":"2024-09-16T11:15:17.299412+0000","last_fullsized":"2024-09-16T11:15:17.299412+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:25.607887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304390+0000","last_change":"2024-09-16T11:15:17.304636+0000","last_active":"2024-09-16T11:15:17.304390+0000","last_peered":"2024-09-16T11:15:17.304390+0000","last_clean":"2024-09-16T11:15:17.304390+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:15:17.304390+0000","last_undegraded":"2024-09-16T11:15:17.304390+0000","last_fullsized":"2024-09-16T11:15:17.304390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:15.280839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":79,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T11:15:16.298776+0000","last_change":"2024-09-16T11:15:10.226884+0000","last_active":"2024-09-16T11:15:16.298776+0000","last_peered":"2024-09-16T11:15:16.298776+0000","last_clean":"2024-09-16T11:15:16.298776+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:15:16.298776+0000","last_undegraded":"2024-09-16T11:15:16.298776+0000","last_fullsized":"2024-09-16T11:15:16.298776+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_clean_scrub_stamp":"2024-09-16T11:15:09.219587+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:56.273706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301644,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070196,"statfs":{"total":96636764160,"available":96327880704,"internally_reserved":0,"allocated":131072,"data_stored":38175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302228,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069612,"statfs":{"total":96636764160,"available":96327282688,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302224,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069616,"statfs":{"total":96636764160,"available":96327286784,"internally_reserved":0,"allocated":724992,"data_stored":627801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:15:20.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:15:20.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:20.646 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:15:20.657 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T11:15:19.009016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":906096,"kb_used_data":1548,"kb_used_omap":4,"kb_used_meta":904507,"kb_avail":282209424,"statfs":{"total":289910292480,"available":288982450176,"internally_reserved":0,"allocated":1585152,"data_stored":1294538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":926215176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.719177"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304322+0000","last_change":"2024-09-16T11:15:17.304545+0000","last_active":"2024-09-16T11:15:17.304322+0000","last_peered":"2024-09-16T11:15:17.304322+0000","last_clean":"2024-09-16T11:15:17.304322+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:15:17.304322+0000","last_undegraded":"2024-09-16T11:15:17.304322+0000","last_fullsized":"2024-09-16T11:15:17.304322+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:52.627160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304116+0000","last_change":"2024-09-16T11:15:17.304329+0000","last_active":"2024-09-16T11:15:17.304116+0000","last_peered":"2024-09-16T11:15:17.304116+0000","last_clean":"2024-09-16T11:15:17.304116+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:15:17.304116+0000","last_undegraded":"2024-09-16T11:15:17.304116+0000","last_fullsized":"2024-09-16T11:15:17.304116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:28.342868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304002+0000","last_change":"2024-09-16T11:15:17.304273+0000","last_active":"2024-09-16T11:15:17.304002+0000","last_peered":"2024-09-16T11:15:17.304002+0000","last_clean":"2024-09-16T11:15:17.304002+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:15:17.304002+0000","last_undegraded":"2024-09-16T11:15:17.304002+0000","last_fullsized":"2024-09-16T11:15:17.304002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:52.193450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304628+0000","last_change":"2024-09-16T11:15:17.304918+0000","last_active":"2024-09-16T11:15:17.304628+0000","last_peered":"2024-09-16T11:15:17.304628+0000","last_clean":"2024-09-16T11:15:17.304628+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:15:17.304628+0000","last_undegraded":"2024-09-16T11:15:17.304628+0000","last_fullsized":"2024-09-16T11:15:17.304628+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:24:48.683341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.302378+0000","last_change":"2024-09-16T11:15:17.302378+0000","last_active":"2024-09-16T11:15:17.302378+0000","last_peered":"2024-09-16T11:15:17.302378+0000","last_clean":"2024-09-16T11:15:17.302378+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:15:17.302378+0000","last_undegraded":"2024-09-16T11:15:17.302378+0000","last_fullsized":"2024-09-16T11:15:17.302378+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.265636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.299500+0000","last_change":"2024-09-16T11:15:17.299727+0000","last_active":"2024-09-16T11:15:17.299500+0000","last_peered":"2024-09-16T11:15:17.299500+0000","last_clean":"2024-09-16T11:15:17.299500+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:15:17.299500+0000","last_undegraded":"2024-09-16T11:15:17.299500+0000","last_fullsized":"2024-09-16T11:15:17.299500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:10.958157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.299412+0000","last_change":"2024-09-16T11:15:17.299646+0000","last_active":"2024-09-16T11:15:17.299412+0000","last_peered":"2024-09-16T11:15:17.299412+0000","last_clean":"2024-09-16T11:15:17.299412+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:15:17.299412+0000","last_undegraded":"2024-09-16T11:15:17.299412+0000","last_fullsized":"2024-09-16T11:15:17.299412+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:25.607887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:17.304390+0000","last_change":"2024-09-16T11:15:17.304636+0000","last_active":"2024-09-16T11:15:17.304390+0000","last_peered":"2024-09-16T11:15:17.304390+0000","last_clean":"2024-09-16T11:15:17.304390+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:15:17.304390+0000","last_undegraded":"2024-09-16T11:15:17.304390+0000","last_fullsized":"2024-09-16T11:15:17.304390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:15.280839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":79,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T11:15:16.298776+0000","last_change":"2024-09-16T11:15:10.226884+0000","last_active":"2024-09-16T11:15:16.298776+0000","last_peered":"2024-09-16T11:15:16.298776+0000","last_clean":"2024-09-16T11:15:16.298776+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:15:16.298776+0000","last_undegraded":"2024-09-16T11:15:16.298776+0000","last_fullsized":"2024-09-16T11:15:16.298776+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_clean_scrub_stamp":"2024-09-16T11:15:09.219587+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:56.273706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301644,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070196,"statfs":{"total":96636764160,"available":96327880704,"internally_reserved":0,"allocated":131072,"data_stored":38175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302228,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069612,"statfs":{"total":96636764160,"available":96327282688,"internally_reserved":0,"allocated":729088,"data_stored":628562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302224,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069616,"statfs":{"total":96636764160,"available":96327286784,"internally_reserved":0,"allocated":724992,"data_stored":627801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:15:20.659 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T11:15:20.659 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-16T11:15:20.659 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-16T11:15:20.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-16T11:15:20.966 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:20.966 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T11:15:20.979 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-16T11:15:20.979 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T11:15:20.988 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-16T11:15:20.988 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T11:15:20.989 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907167', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGqAFr/poL95sO4gd9TCrLiYuCVqFzL5efo5srdCOpECOtvr6+2wExCxiQcB5UD5JOjszyCNZqaCh2dYXBmp8Y=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIn6j46Ue+ntpffguTLbpcQNYJduNZD9odWBx7pmD0KsupCXWd35E+7vLdXpX1BFhsiUSaYof4nIDQ6cVd+aL2M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T11:15:20.989 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T11:15:20.992 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T11:15:21.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:15:21.048 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:15:21.081 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:21.081 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T11:15:21.139 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T11:15:21.140 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 11:14:39.157603547 +0000 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 11:11:53.649666911 +0000 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 11:11:53.649666911 +0000 2024-09-16T11:15:21.197 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 11:11:53.649666911 +0000 2024-09-16T11:15:21.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T11:15:21.263 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T11:15:21.263 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T11:15:21.263 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000283959 s, 1.8 MB/s 2024-09-16T11:15:21.265 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T11:15:21.324 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-09-16T11:15:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T11:15:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:15:21.381 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T11:15:21.381 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:15:21.381 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:15:21.381 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 11:14:39.157603547 +0000 2024-09-16T11:15:21.382 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 11:11:53.981660541 +0000 2024-09-16T11:15:21.382 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 11:11:53.981660541 +0000 2024-09-16T11:15:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 11:11:53.981660541 +0000 2024-09-16T11:15:21.382 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T11:15:21.446 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T11:15:21.446 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T11:15:21.446 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000174345 s, 2.9 MB/s 2024-09-16T11:15:21.447 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T11:15:21.503 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 11:14:39.158603527 +0000 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 11:11:54.300654421 +0000 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 11:11:54.300654421 +0000 2024-09-16T11:15:21.559 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 11:11:54.300654421 +0000 2024-09-16T11:15:21.559 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T11:15:21.623 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T11:15:21.623 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T11:15:21.623 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000207434 s, 2.5 MB/s 2024-09-16T11:15:21.625 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T11:15:21.680 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-16 11:14:39.158603527 +0000 2024-09-16T11:15:21.736 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-16 11:11:54.624648205 +0000 2024-09-16T11:15:21.737 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-16 11:11:54.624648205 +0000 2024-09-16T11:15:21.737 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-16 11:11:54.624648205 +0000 2024-09-16T11:15:21.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T11:15:21.800 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-16T11:15:21.800 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-16T11:15:21.800 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00019101 s, 2.7 MB/s 2024-09-16T11:15:21.801 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T11:15:21.858 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:15:21.858 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:15:21.858 INFO:tasks.ceph:Generating config... 2024-09-16T11:15:21.859 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:21.860 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:21.861 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T11:15:21.861 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T11:15:21.861 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T11:15:21.861 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T11:15:21.861 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T11:15:21.861 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T11:15:21.861 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T11:15:21.861 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T11:15:21.861 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T11:15:21.861 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T11:15:21.861 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T11:15:21.861 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T11:15:21.861 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T11:15:21.861 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T11:15:21.862 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T11:15:21.862 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-16T11:15:21.862 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-16T11:15:21.947 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.keyring 2024-09-16T11:15:21.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-16T11:15:21.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T11:15:22.054 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.103')] 2024-09-16T11:15:22.054 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.103', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T11:15:22.055 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:22.055 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:15:22.109 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.103 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: generated fsid 6599c54d-59b2-4c65-8b6b-ee47e617df73 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = octopus 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 0 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:fsid 6599c54d-59b2-4c65-8b6b-ee47e617df73 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:last_changed 2024-09-16T11:15:22.185857+0000 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-09-16T11:15:22.185857+0000 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:min_mon_release 15 (octopus) 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:election_strategy: 1 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-09-16T11:15:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-16T11:15:22.188 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:15:22.243 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 6599c54d-59b2-4c65-8b6b-ee47e617df73... 2024-09-16T11:15:22.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T11:15:22.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T11:15:22.299 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-09-16T11:15:22.299 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-09-16T11:15:22.299 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-09-16T11:15:22.300 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-09-16T11:15:22.301 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.302 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-09-16T11:15:22.302 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-09-16T11:15:22.302 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:15:22.302 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:15:22.302 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-09-16T11:15:22.302 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:15:22.303 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-09-16T11:15:22.303 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:15:22.303 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:15:22.303 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-09-16T11:15:22.303 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.303 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-09-16T11:15:22.304 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = 172.21.15.103 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: mon client directed command retry = 5 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on pool no app = False 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: ms inject socket failures = 5000 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 6599c54d-59b2-4c65-8b6b-ee47e617df73 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.305 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.306 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.307 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-09-16T11:15:22.308 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-09-16T11:15:22.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.308 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:15:22.308 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.308 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.309 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-09-16T11:15:22.310 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-09-16T11:15:22.310 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-09-16T11:15:22.310 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-09-16T11:15:22.310 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = bitmap 2024-09-16T11:15:22.310 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-09-16T11:15:22.310 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-09-16T11:15:22.311 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-09-16T11:15:22.312 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:15:22.313 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-09-16T11:15:22.314 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd = 20 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: debug rbd_mirror = 15 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: log to stderr = False 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: rbd default features = 125 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: rbd mirror image policy type = simple 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.0] 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.315 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.1] 2024-09-16T11:15:22.316 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.316 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.316 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.2] 2024-09-16T11:15:22.316 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.316 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.317 INFO:teuthology.orchestra.run.smithi001.stdout:[client.mirror.3] 2024-09-16T11:15:22.317 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.317 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.326 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-09-16T11:15:22.326 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-09-16T11:15:22.326 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:15:22.326 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-09-16T11:15:22.326 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:15:22.327 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-09-16T11:15:22.328 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-09-16T11:15:22.329 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = 172.21.15.103 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: mon client directed command retry = 5 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on pool no app = False 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: ms inject socket failures = 5000 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 6599c54d-59b2-4c65-8b6b-ee47e617df73 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-09-16T11:15:22.330 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-09-16T11:15:22.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.331 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-09-16T11:15:22.331 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-09-16T11:15:22.331 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:15:22.331 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.331 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-09-16T11:15:22.332 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.333 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: bdev async discard = True 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: bdev enable discard = True 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore allocator = bitmap 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 1/20 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 1/20 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 4/10 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T11:15:22.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-09-16T11:15:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.337 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-09-16T11:15:22.338 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: debug rbd = 20 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: debug rbd_mirror = 15 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: log to stderr = False 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: rbd default features = 125 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: rbd mirror image policy type = simple 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.0] 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.339 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.1] 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.2] 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.3] 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:15:22.340 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:15:22.341 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-16T11:15:22.341 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-16T11:15:22.431 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T11:15:22.431 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:22.431 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-16T11:15:22.446 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:22.446 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-16T11:15:22.501 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:15:22.502 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:15:22.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T11:15:22.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T11:15:22.539 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:15:22.539 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:22.595 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:15:22.595 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:22.595 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T11:15:22.595 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T11:15:22.630 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:22.630 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:22.684 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T11:15:22.685 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T11:15:22.685 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T11:15:22.735 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T11:15:22.737 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T11:15:22.738 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T11:15:22.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-16T11:15:22.776 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-16T11:15:22.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T11:15:22.828 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T11:15:22.841 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T11:15:22.917 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T11:15:22.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T11:15:23.006 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T11:15:23.019 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T11:15:23.097 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T11:15:23.109 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T11:15:23.186 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T11:15:23.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T11:15:23.276 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T11:15:23.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T11:15:23.365 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T11:15:23.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T11:15:23.454 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T11:15:23.466 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T11:15:23.467 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:15:23.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-16T11:15:23.531 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:15:23.531 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-16T11:15:23.531 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:15:23.531 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:15:23.604 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:15:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:15:23.605 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:15:24.116 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T11:15:24.118 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T11:15:24.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-16T11:15:24.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-16T11:15:24.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-16T11:15:24.327 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:15:24.327 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-16T11:15:24.327 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:15:24.327 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:15:24.400 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:15:24.401 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:15:24.864 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T11:15:24.866 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T11:15:24.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-16T11:15:24.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-16T11:15:24.987 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-16T11:15:25.053 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:15:25.053 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-16T11:15:25.053 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:15:25.053 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:15:25.126 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:15:25.127 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:15:25.627 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-09-16T11:15:25.629 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-09-16T11:15:25.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-16T11:15:25.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-16T11:15:25.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:25.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:25.887+0000 7f0ee8a0c440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-16T11:15:25.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:25.893+0000 7f0ee8a0c440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-16T11:15:25.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:25.893+0000 7f0ee8a0c440 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T11:15:25.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:25.893+0000 7f0ee8a0c440 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T11:15:25.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:25.893+0000 7f0ee8a0c440 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T11:15:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:25.896+0000 7f0ee8a0c440 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-16T11:15:26.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:26.421+0000 7f0ee8a0c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:27.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:27.189+0000 7f0ee8a0c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:27.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:27.957+0000 7f0ee8a0c440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:28.764 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-16T11:15:28.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:28.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:28.880+0000 7f2001125440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-16T11:15:28.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:28.880+0000 7f2001125440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-16T11:15:28.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:28.880+0000 7f2001125440 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T11:15:28.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:28.881+0000 7f2001125440 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T11:15:28.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:28.881+0000 7f2001125440 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T11:15:28.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:28.881+0000 7f2001125440 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-16T11:15:29.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:29.408+0000 7f2001125440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:30.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:30.176+0000 7f2001125440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:30.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:30.938+0000 7f2001125440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:31.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-16T11:15:31.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:31.854+0000 7fdcea91d440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-16T11:15:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:31.855+0000 7fdcea91d440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-16T11:15:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:31.855+0000 7fdcea91d440 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T11:15:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:31.855+0000 7fdcea91d440 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T11:15:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:31.855+0000 7fdcea91d440 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T11:15:31.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:31.856+0000 7fdcea91d440 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-16T11:15:32.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:32.381+0000 7fdcea91d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:33.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:33.145+0000 7fdcea91d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:33.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-09-16T11:15:33.912+0000 7fdcea91d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T11:15:34.708 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-16T11:15:34.737 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T11:15:34.737 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:15:34.737 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-16T11:15:34.769 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:34.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-16T11:15:34.805 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:34.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-16T11:15:34.874 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:34.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-16T11:15:34.941 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-09-16T11:15:34.941 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-16T11:15:34.961 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:34.961 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-16T11:15:34.998 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:34.999 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-16T11:15:35.054 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:35.054 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-16T11:15:35.110 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:35.110 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-16T11:15:35.166 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:35.166 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-16T11:15:35.222 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:35.222 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-16T11:15:35.278 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:35.278 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-16T11:15:35.334 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:35.334 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-16T11:15:35.389 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T11:15:35.390 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-16T11:15:35.452 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-09-16T11:15:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBKE+hmoNzMKxAAmSHIrf8JbRP9ZsA3uCBUWg== 2024-09-16T11:15:35.452 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-09-16T11:15:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBNE+hmHDcHNRAAcmO4RLU0eiTmkGs4iThHsw== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBQE+hm1FKINBAAidwrytZ3yyDL9pnaRaumuw== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBTE+hmwjAAMxAAM3I/Xfu8/Hy+uqTnIs04Lg== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBKE+hmjkBALhAA7wH19UpvqhDd/zwvkN9CFA== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror] 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBKE+hmxQ1VMRAA9uclWtEEcw90G06sliO3Xw== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.0] 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBKE+hmnjCkNhAAvRiTPTzh61ciUfawTYCXQQ== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.1] 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLE+hmuGxcABAADC58xuy/p8gUEf21wBnUlA== 2024-09-16T11:15:35.453 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.2] 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLE+hmmpy+BRAAP5h7Mkbvx8eLvCi4JzIhPg== 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.3] 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLE+hmyJYXCxAAdrvC9gp7rFatwVWh4xD1VA== 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.4] 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLE+hmgPhwEBAALY1d/h9oOPlphtn0eTXCnQ== 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.5] 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLE+hmJAS/FRAAy0vpjmzh3SuTmJf2Ct70lA== 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout:[client.mirror.6] 2024-09-16T11:15:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLE+hmQvwOGxAA33B/pv3Hq3lTEpxpwqkZpQ== 2024-09-16T11:15:35.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T11:15:35.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:15:35.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:15:35.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:15:35.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:35.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:35.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:35.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:35.876 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:35.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:35.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:36.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:36.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:15:36.125 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T11:15:36.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-16T11:15:36.150 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-16T11:15:36.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-16T11:15:36.265 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:15:36.320 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-16T11:15:36.321 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-16T11:15:36.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-16T11:15:36.362 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-16T11:15:36.363 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-16T11:15:36.363 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-16T11:15:36.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-16T11:15:36.366 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-16T11:15:36.366 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T11:15:36.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-16T11:15:36.528 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2024-09-16T11:15:36.530 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:36.528+0000 7f57dbfc3ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T11:15:36.539 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-16T11:15:36.539 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:36.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-16T11:15:36.564 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:36.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-16T11:15:36.572 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:36.570+0000 7f57dbfc3ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T11:15:36.621 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:36.619+0000 7f57dbfc3ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T11:15:36.629 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:15:36.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-16T11:15:36.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster cluster2 osd new ccbdd57e-4ebb-4834-b138-dce5b7f305da 0 2024-09-16T11:15:36.715 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:36.713+0000 7f57dbfc3ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:15:36.809 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:36.807+0000 7f57dbfc3ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T11:15:36.900 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-09-16T11:15:36.910 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster cluster2 osd new e86d8bdd-47fb-438e-b101-4b10cd95fac6 1 2024-09-16T11:15:37.046 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:37.044+0000 7f57dbfc3ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T11:15:37.075 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-09-16T11:15:37.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster cluster2 osd new e5c3a350-1028-4bca-a644-b4144e776f7f 2 2024-09-16T11:15:37.091 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:37.089+0000 7f57dbfc3ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T11:15:37.194 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:37.192+0000 7f57dbfc3ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T11:15:37.250 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-09-16T11:15:37.261 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-16T11:15:37.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-16T11:15:37.263 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-16T11:15:37.263 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-16T11:15:37.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-16T11:15:37.265 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-16T11:15:37.266 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-16T11:15:37.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-16T11:15:37.268 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-16T11:15:37.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:15:37.630 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:37.631 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":5,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:15:37.248420+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:37.641 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-16T11:15:37.641 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T11:15:37.858 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:15:37.856+0000 7f48bf43b440 -1 Falling back to public interface 2024-09-16T11:15:37.863 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:15:37.861+0000 7f1e3212a440 -1 Falling back to public interface 2024-09-16T11:15:37.864 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:15:37.862+0000 7f82cad24440 -1 Falling back to public interface 2024-09-16T11:15:37.905 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:37.903+0000 7f57dbfc3ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T11:15:37.944 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:15:37.950 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:37.948+0000 7f57dbfc3ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T11:15:38.092 INFO:teuthology.misc.health.smithi103.stdout: 2024-09-16T11:15:38.092 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":5,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:15:37.248420+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:38.101 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T11:15:38.276 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.274+0000 7f57dbfc3ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T11:15:38.359 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.357+0000 7f57dbfc3ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T11:15:38.452 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.450+0000 7f57dbfc3ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:15:38.495 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.493+0000 7f57dbfc3ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T11:15:38.591 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.589+0000 7f57dbfc3ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T11:15:38.644 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.642+0000 7f57dbfc3ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T11:15:38.685 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.683+0000 7f57dbfc3ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T11:15:38.782 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.780+0000 7f57dbfc3ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T11:15:38.832 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.829+0000 7f57dbfc3ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T11:15:38.880 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.878+0000 7f57dbfc3ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T11:15:38.934 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.932+0000 7f57dbfc3ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T11:15:38.976 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:38.974+0000 7f57dbfc3ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T11:15:39.018 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:39.016+0000 7f57dbfc3ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T11:15:39.586 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:39.584+0000 7f57dbfc3ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T11:15:40.594 INFO:tasks.ceph.cluster2.mgr.x.smithi001.stderr:2024-09-16T11:15:40.592+0000 7f57c051a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T11:15:41.339 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:15:41.338+0000 7f48bf43b440 -1 osd.0 0 log_to_monitors true 2024-09-16T11:15:41.340 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:15:41.338+0000 7f1e3212a440 -1 osd.2 0 log_to_monitors true 2024-09-16T11:15:41.366 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:15:41.364+0000 7f82cad24440 -1 osd.1 0 log_to_monitors true 2024-09-16T11:15:44.243 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:15:44.242+0000 7f48bbdf9640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T11:15:44.247 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:15:44.246+0000 7f48b3484640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:44.269 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:15:44.267+0000 7f1e2cbf6640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T11:15:44.271 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:15:44.270+0000 7f1e25a84640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:44.312 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:15:44.310+0000 7f82c57f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T11:15:44.317 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:15:44.316+0000 7f82be684640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:44.321 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:15:44.319+0000 7f82b9b43640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:15:44.403 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:15:44.654 INFO:teuthology.misc.health.smithi103.stdout: 2024-09-16T11:15:44.655 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":7,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:15:42.592660+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:44.665 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T11:15:50.969 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:15:51.231 INFO:teuthology.misc.health.smithi103.stdout: 2024-09-16T11:15:51.231 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":12,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:15:49.248365+0000","last_up_change":"2024-09-16T11:15:45.243714+0000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:45.620615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":32440},{"type":"v1","addr":"172.21.15.103:6801","nonce":32440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":32440},{"type":"v1","addr":"172.21.15.103:6803","nonce":32440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":32440},{"type":"v1","addr":"172.21.15.103:6807","nonce":32440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":32440},{"type":"v1","addr":"172.21.15.103:6805","nonce":32440}]},"public_addr":"172.21.15.103:6801/32440","cluster_addr":"172.21.15.103:6803/32440","heartbeat_back_addr":"172.21.15.103:6807/32440","heartbeat_front_addr":"172.21.15.103:6805/32440","state":["exists","up"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":32441},{"type":"v1","addr":"172.21.15.103:6815","nonce":32441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":32441},{"type":"v1","addr":"172.21.15.103:6819","nonce":32441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":32441},{"type":"v1","addr":"172.21.15.103:6823","nonce":32441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":32441},{"type":"v1","addr":"172.21.15.103:6821","nonce":32441}]},"public_addr":"172.21.15.103:6815/32441","cluster_addr":"172.21.15.103:6819/32441","heartbeat_back_addr":"172.21.15.103:6823/32441","heartbeat_front_addr":"172.21.15.103:6821/32441","state":["exists","up"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":32442},{"type":"v1","addr":"172.21.15.103:6809","nonce":32442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":32442},{"type":"v1","addr":"172.21.15.103:6811","nonce":32442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":32442},{"type":"v1","addr":"172.21.15.103:6818","nonce":32442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":32442},{"type":"v1","addr":"172.21.15.103:6814","nonce":32442}]},"public_addr":"172.21.15.103:6809/32442","cluster_addr":"172.21.15.103:6811/32442","heartbeat_back_addr":"172.21.15.103:6818/32442","heartbeat_front_addr":"172.21.15.103:6814/32442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.330275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.341245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.390116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:51.242 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T11:15:51.243 INFO:tasks.ceph:Creating RBD pool 2024-09-16T11:15:51.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-16T11:15:52.266 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'rbd' created 2024-09-16T11:15:52.283 DEBUG:teuthology.orchestra.run.smithi103:> rbd --cluster cluster2 pool init rbd 2024-09-16T11:15:55.287 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-16T11:15:55.288 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-16T11:15:55.288 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-16T11:15:55.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-16T11:15:55.289 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T11:15:55.601 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:55.612 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":36500},{"type":"v1","addr":"172.21.15.1:6827","nonce":36500}]},"active_addr":"172.21.15.1:6827/36500","active_change":"2024-09-16T11:15:39.586565+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3034364225}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":1929775569}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2331336929}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":1602197068}]}]} 2024-09-16T11:15:55.614 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-16T11:15:55.615 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-16T11:15:55.615 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:15:55.877 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:55.877 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":16,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:15:55.273206+0000","last_up_change":"2024-09-16T11:15:45.243714+0000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:45.620615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:51.517749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":32440},{"type":"v1","addr":"172.21.15.103:6801","nonce":32440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":32440},{"type":"v1","addr":"172.21.15.103:6803","nonce":32440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":32440},{"type":"v1","addr":"172.21.15.103:6807","nonce":32440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":32440},{"type":"v1","addr":"172.21.15.103:6805","nonce":32440}]},"public_addr":"172.21.15.103:6801/32440","cluster_addr":"172.21.15.103:6803/32440","heartbeat_back_addr":"172.21.15.103:6807/32440","heartbeat_front_addr":"172.21.15.103:6805/32440","state":["exists","up"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":32441},{"type":"v1","addr":"172.21.15.103:6815","nonce":32441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":32441},{"type":"v1","addr":"172.21.15.103:6819","nonce":32441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":32441},{"type":"v1","addr":"172.21.15.103:6823","nonce":32441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":32441},{"type":"v1","addr":"172.21.15.103:6821","nonce":32441}]},"public_addr":"172.21.15.103:6815/32441","cluster_addr":"172.21.15.103:6819/32441","heartbeat_back_addr":"172.21.15.103:6823/32441","heartbeat_front_addr":"172.21.15.103:6821/32441","state":["exists","up"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":32442},{"type":"v1","addr":"172.21.15.103:6809","nonce":32442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":32442},{"type":"v1","addr":"172.21.15.103:6811","nonce":32442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":32442},{"type":"v1","addr":"172.21.15.103:6818","nonce":32442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":32442},{"type":"v1","addr":"172.21.15.103:6814","nonce":32442}]},"public_addr":"172.21.15.103:6809/32442","cluster_addr":"172.21.15.103:6811/32442","heartbeat_back_addr":"172.21.15.103:6818/32442","heartbeat_front_addr":"172.21.15.103:6814/32442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.330275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.341245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.390116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:55.889 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-16T11:15:55.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:15:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:56.147 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":16,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:15:55.273206+0000","last_up_change":"2024-09-16T11:15:45.243714+0000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:45.620615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:51.517749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":32440},{"type":"v1","addr":"172.21.15.103:6801","nonce":32440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":32440},{"type":"v1","addr":"172.21.15.103:6803","nonce":32440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":32440},{"type":"v1","addr":"172.21.15.103:6807","nonce":32440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":32440},{"type":"v1","addr":"172.21.15.103:6805","nonce":32440}]},"public_addr":"172.21.15.103:6801/32440","cluster_addr":"172.21.15.103:6803/32440","heartbeat_back_addr":"172.21.15.103:6807/32440","heartbeat_front_addr":"172.21.15.103:6805/32440","state":["exists","up"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":32441},{"type":"v1","addr":"172.21.15.103:6815","nonce":32441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":32441},{"type":"v1","addr":"172.21.15.103:6819","nonce":32441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":32441},{"type":"v1","addr":"172.21.15.103:6823","nonce":32441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":32441},{"type":"v1","addr":"172.21.15.103:6821","nonce":32441}]},"public_addr":"172.21.15.103:6815/32441","cluster_addr":"172.21.15.103:6819/32441","heartbeat_back_addr":"172.21.15.103:6823/32441","heartbeat_front_addr":"172.21.15.103:6821/32441","state":["exists","up"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":32442},{"type":"v1","addr":"172.21.15.103:6809","nonce":32442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":32442},{"type":"v1","addr":"172.21.15.103:6811","nonce":32442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":32442},{"type":"v1","addr":"172.21.15.103:6818","nonce":32442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":32442},{"type":"v1","addr":"172.21.15.103:6814","nonce":32442}]},"public_addr":"172.21.15.103:6809/32442","cluster_addr":"172.21.15.103:6811/32442","heartbeat_back_addr":"172.21.15.103:6818/32442","heartbeat_front_addr":"172.21.15.103:6814/32442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.330275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.341245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.390116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:15:56.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-16T11:15:56.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-16T11:15:56.160 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-16T11:15:56.291 INFO:teuthology.orchestra.run.smithi103.stdout:34359738374 2024-09-16T11:15:56.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T11:15:56.295 INFO:teuthology.orchestra.run.smithi103.stdout:34359738374 2024-09-16T11:15:56.295 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T11:15:56.296 INFO:teuthology.orchestra.run.smithi103.stdout:34359738374 2024-09-16T11:15:56.296 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T11:15:56.605 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2024-09-16T11:15:56.610 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2024-09-16T11:15:56.619 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0 2024-09-16T11:15:56.622 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.2 2024-09-16T11:15:56.645 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2024-09-16T11:15:56.657 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-16T11:15:57.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T11:15:57.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T11:15:57.658 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T11:15:57.918 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2024-09-16T11:15:57.930 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.0 2024-09-16T11:15:57.937 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2024-09-16T11:15:57.950 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-16T11:15:57.961 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2024-09-16T11:15:57.973 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.2 2024-09-16T11:15:58.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T11:15:58.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T11:15:58.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T11:15:59.233 INFO:teuthology.orchestra.run.smithi103.stdout:34359738374 2024-09-16T11:15:59.245 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2024-09-16T11:15:59.246 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:59.290 INFO:teuthology.orchestra.run.smithi103.stdout:34359738374 2024-09-16T11:15:59.296 INFO:teuthology.orchestra.run.smithi103.stdout:34359738374 2024-09-16T11:15:59.303 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-09-16T11:15:59.303 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:59.309 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2024-09-16T11:15:59.309 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:59.309 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T11:15:59.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:15:59.606 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:15:59.617 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T11:15:59.593678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893668,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221852,"statfs":{"total":289910292480,"available":288995176448,"internally_reserved":0,"allocated":1613824,"data_stored":1303624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.326290"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281740+0000","last_change":"2024-09-16T11:15:55.281946+0000","last_active":"2024-09-16T11:15:55.281740+0000","last_peered":"2024-09-16T11:15:55.281740+0000","last_clean":"2024-09-16T11:15:55.281740+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:15:55.281740+0000","last_undegraded":"2024-09-16T11:15:55.281740+0000","last_fullsized":"2024-09-16T11:15:55.281740+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:39.247359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281789+0000","last_change":"2024-09-16T11:15:55.282001+0000","last_active":"2024-09-16T11:15:55.281789+0000","last_peered":"2024-09-16T11:15:55.281789+0000","last_clean":"2024-09-16T11:15:55.281789+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:15:55.281789+0000","last_undegraded":"2024-09-16T11:15:55.281789+0000","last_fullsized":"2024-09-16T11:15:55.281789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:58.677788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281400+0000","last_change":"2024-09-16T11:15:55.281660+0000","last_active":"2024-09-16T11:15:55.281400+0000","last_peered":"2024-09-16T11:15:55.281400+0000","last_clean":"2024-09-16T11:15:55.281400+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:15:55.281400+0000","last_undegraded":"2024-09-16T11:15:55.281400+0000","last_fullsized":"2024-09-16T11:15:55.281400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:18.641643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281352+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:15:55.281352+0000","last_peered":"2024-09-16T11:15:55.281352+0000","last_clean":"2024-09-16T11:15:55.281352+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:15:55.281352+0000","last_undegraded":"2024-09-16T11:15:55.281352+0000","last_fullsized":"2024-09-16T11:15:55.281352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:28.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.279290+0000","last_change":"2024-09-16T11:15:55.279290+0000","last_active":"2024-09-16T11:15:55.279290+0000","last_peered":"2024-09-16T11:15:55.279290+0000","last_clean":"2024-09-16T11:15:55.279290+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:15:55.279290+0000","last_undegraded":"2024-09-16T11:15:55.279290+0000","last_fullsized":"2024-09-16T11:15:55.279290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:09.009156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.276133+0000","last_change":"2024-09-16T11:15:55.276334+0000","last_active":"2024-09-16T11:15:55.276133+0000","last_peered":"2024-09-16T11:15:55.276133+0000","last_clean":"2024-09-16T11:15:55.276133+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:15:55.276133+0000","last_undegraded":"2024-09-16T11:15:55.276133+0000","last_fullsized":"2024-09-16T11:15:55.276133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:08.820896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.276163+0000","last_change":"2024-09-16T11:15:55.276351+0000","last_active":"2024-09-16T11:15:55.276163+0000","last_peered":"2024-09-16T11:15:55.276163+0000","last_clean":"2024-09-16T11:15:55.276163+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:15:55.276163+0000","last_undegraded":"2024-09-16T11:15:55.276163+0000","last_fullsized":"2024-09-16T11:15:55.276163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:33.264611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.282082+0000","last_change":"2024-09-16T11:15:55.282319+0000","last_active":"2024-09-16T11:15:55.282082+0000","last_peered":"2024-09-16T11:15:55.282082+0000","last_clean":"2024-09-16T11:15:55.282082+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:15:55.282082+0000","last_undegraded":"2024-09-16T11:15:55.282082+0000","last_fullsized":"2024-09-16T11:15:55.282082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:46.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":81,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:54.282334+0000","last_change":"2024-09-16T11:15:48.254170+0000","last_active":"2024-09-16T11:15:54.282334+0000","last_peered":"2024-09-16T11:15:54.282334+0000","last_clean":"2024-09-16T11:15:54.282334+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:15:54.282334+0000","last_undegraded":"2024-09-16T11:15:54.282334+0000","last_fullsized":"2024-09-16T11:15:54.282334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_clean_scrub_stamp":"2024-09-16T11:15:46.244458+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:31.333661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":139264,"data_stored":40950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":737280,"data_stored":631337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":737280,"data_stored":631337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:15:59.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:15:59.873 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:15:59.873 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:15:59.884 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T11:15:59.593678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893668,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221852,"statfs":{"total":289910292480,"available":288995176448,"internally_reserved":0,"allocated":1613824,"data_stored":1303624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.326290"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281740+0000","last_change":"2024-09-16T11:15:55.281946+0000","last_active":"2024-09-16T11:15:55.281740+0000","last_peered":"2024-09-16T11:15:55.281740+0000","last_clean":"2024-09-16T11:15:55.281740+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:15:55.281740+0000","last_undegraded":"2024-09-16T11:15:55.281740+0000","last_fullsized":"2024-09-16T11:15:55.281740+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:39.247359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281789+0000","last_change":"2024-09-16T11:15:55.282001+0000","last_active":"2024-09-16T11:15:55.281789+0000","last_peered":"2024-09-16T11:15:55.281789+0000","last_clean":"2024-09-16T11:15:55.281789+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:15:55.281789+0000","last_undegraded":"2024-09-16T11:15:55.281789+0000","last_fullsized":"2024-09-16T11:15:55.281789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:58.677788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281400+0000","last_change":"2024-09-16T11:15:55.281660+0000","last_active":"2024-09-16T11:15:55.281400+0000","last_peered":"2024-09-16T11:15:55.281400+0000","last_clean":"2024-09-16T11:15:55.281400+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:15:55.281400+0000","last_undegraded":"2024-09-16T11:15:55.281400+0000","last_fullsized":"2024-09-16T11:15:55.281400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:18.641643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.281352+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:15:55.281352+0000","last_peered":"2024-09-16T11:15:55.281352+0000","last_clean":"2024-09-16T11:15:55.281352+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:15:55.281352+0000","last_undegraded":"2024-09-16T11:15:55.281352+0000","last_fullsized":"2024-09-16T11:15:55.281352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:28.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.279290+0000","last_change":"2024-09-16T11:15:55.279290+0000","last_active":"2024-09-16T11:15:55.279290+0000","last_peered":"2024-09-16T11:15:55.279290+0000","last_clean":"2024-09-16T11:15:55.279290+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:15:55.279290+0000","last_undegraded":"2024-09-16T11:15:55.279290+0000","last_fullsized":"2024-09-16T11:15:55.279290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:09.009156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.276133+0000","last_change":"2024-09-16T11:15:55.276334+0000","last_active":"2024-09-16T11:15:55.276133+0000","last_peered":"2024-09-16T11:15:55.276133+0000","last_clean":"2024-09-16T11:15:55.276133+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:15:55.276133+0000","last_undegraded":"2024-09-16T11:15:55.276133+0000","last_fullsized":"2024-09-16T11:15:55.276133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:08.820896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.276163+0000","last_change":"2024-09-16T11:15:55.276351+0000","last_active":"2024-09-16T11:15:55.276163+0000","last_peered":"2024-09-16T11:15:55.276163+0000","last_clean":"2024-09-16T11:15:55.276163+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:15:55.276163+0000","last_undegraded":"2024-09-16T11:15:55.276163+0000","last_fullsized":"2024-09-16T11:15:55.276163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:33.264611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:15:55.282082+0000","last_change":"2024-09-16T11:15:55.282319+0000","last_active":"2024-09-16T11:15:55.282082+0000","last_peered":"2024-09-16T11:15:55.282082+0000","last_clean":"2024-09-16T11:15:55.282082+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:15:55.282082+0000","last_undegraded":"2024-09-16T11:15:55.282082+0000","last_fullsized":"2024-09-16T11:15:55.282082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:46.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":81,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:15:54.282334+0000","last_change":"2024-09-16T11:15:48.254170+0000","last_active":"2024-09-16T11:15:54.282334+0000","last_peered":"2024-09-16T11:15:54.282334+0000","last_clean":"2024-09-16T11:15:54.282334+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:15:54.282334+0000","last_undegraded":"2024-09-16T11:15:54.282334+0000","last_fullsized":"2024-09-16T11:15:54.282334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_clean_scrub_stamp":"2024-09-16T11:15:46.244458+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:31.333661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":139264,"data_stored":40950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":737280,"data_stored":631337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":737280,"data_stored":631337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:15:59.885 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T11:15:59.885 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-16T11:15:59.885 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-16T11:15:59.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-16T11:16:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:16:00.174 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T11:16:00.185 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-16T11:16:00.186 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T11:16:00.196 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T11:16:00.196 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:16:00.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:00.527 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror 2024-09-16T11:16:00.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:00.868 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.0 2024-09-16T11:16:00.881 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:01.209 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.1 2024-09-16T11:16:01.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:01.558 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.2 2024-09-16T11:16:01.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:01.897 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.3 2024-09-16T11:16:01.910 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:02.236 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror 2024-09-16T11:16:02.250 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:02.581 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.0 2024-09-16T11:16:02.595 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:02.922 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.1 2024-09-16T11:16:02.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:03.264 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.2 2024-09-16T11:16:03.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:16:03.606 INFO:teuthology.orchestra.run.smithi103.stderr:updated caps for client.mirror.3 2024-09-16T11:16:03.619 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:16:03.630 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:16:03.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:16:03.633 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:16:03.645 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:16:03.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:16:03.647 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T11:16:03.667 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T11:16:03.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T11:16:03.670 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T11:16:03.682 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T11:16:03.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T11:16:03.684 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T11:16:03.701 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T11:16:03.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T11:16:03.703 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T11:16:03.715 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T11:16:03.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T11:16:03.718 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T11:16:03.744 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T11:16:03.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T11:16:03.749 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T11:16:03.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T11:16:03.761 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T11:16:03.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T11:16:03.764 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-16T11:16:03.781 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726485363 2024-09-16T11:16:03.781 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-16T11:16:03.781 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-16T11:16:03.796 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726485363 2024-09-16T11:16:03.796 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-16T11:16:03.797 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T11:16:03.815 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:16:03.815 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T11:16:03.816 DEBUG:teuthology.orchestra.run.smithi103:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:16:03.816 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-16T11:16:03.816 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-16T11:16:03.816 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-16T11:16:03.817 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-16T11:16:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:16:03.832 INFO:teuthology.orchestra.run.smithi103.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-16T11:16:03.832 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:16:03.887 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:16:03.888 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-16T11:16:03.943 INFO:tasks.workunit:timeout=6h 2024-09-16T11:16:03.943 INFO:tasks.workunit:cleanup=True 2024-09-16T11:16:03.943 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:16:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-16T11:16:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: git switch -c 2024-09-16T11:16:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Or undo this operation with: 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: git switch - 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:16:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T11:16:54.002 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-16T11:16:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T11:16:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T11:16:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T11:16:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T11:16:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T11:16:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T11:16:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T11:16:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T11:16:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T11:16:54.968 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-16T11:16:54.968 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-16T11:16:54.984 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-16T11:16:54.985 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-16T11:16:54.985 DEBUG:teuthology.orchestra.run.smithi103:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T11:16:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ IMAGE_COUNT=50 2024-09-16T11:16:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ export LOCKDEP=0 2024-09-16T11:16:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ LOCKDEP=0 2024-09-16T11:16:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T11:16:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ type xmlstarlet 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ XMLSTARLET=xmlstarlet 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CLUSTER1=cluster1 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CLUSTER2=cluster2 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ POOL=mirror 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ PARENT_POOL=mirror_parent 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ NS1=ns1 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ NS2=ns2 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ TEMPDIR= 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ID=mirror 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2024-09-16T11:16:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ MIRROR_POOL_MODE=image 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS='--id mirror' 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T11:16:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-16T11:16:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-16T11:16:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_BIN=. 2024-09-16T11:16:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_SRC=. 2024-09-16T11:16:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 -gt 0 ']' 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ setup 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local c 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ setup_tempdir 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:16:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mktemp -d 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ TEMPDIR=/tmp/tmp.2dE0YG6s4q 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z 1 ']' 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ setup_pools cluster1 cluster2 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local remote_cluster=cluster2 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mon_map_file 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mon_addr 2024-09-16T11:16:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local admin_key_file 2024-09-16T11:16:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local uuid 2024-09-16T11:16:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:16:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-16T11:16:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror' already exists 2024-09-16T11:16:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:16:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-16T11:16:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror_parent' already exists 2024-09-16T11:16:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:16:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-16T11:17:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-16T11:17:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-16T11:17:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-16T11:17:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-16T11:17:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-16T11:17:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-16T11:17:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-16T11:17:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z ']' 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z ']' 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_add cluster1 mirror cluster2 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local client_cluster=cluster2 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local remote_cluster=cluster2 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local uuid_var_name 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local error_code 2024-09-16T11:17:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local peer_uuid 2024-09-16T11:17:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:17:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:17:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-16T11:17:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_uuid=d181ca3d-6a91-467e-82e9-ab82ad78bbd1 2024-09-16T11:17:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 -eq 0 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror_parent 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local client_cluster=cluster2 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local remote_cluster=cluster2 2024-09-16T11:17:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local uuid_var_name 2024-09-16T11:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local error_code 2024-09-16T11:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local peer_uuid 2024-09-16T11:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:17:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-16T11:17:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_uuid=bca9c39f-fa73-4a5b-8569-ef4a0c3ec39a 2024-09-16T11:17:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:17:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 -eq 0 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ setup_pools cluster2 cluster1 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local remote_cluster=cluster1 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mon_map_file 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mon_addr 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local admin_key_file 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local uuid 2024-09-16T11:17:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-16T11:17:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror' already exists 2024-09-16T11:17:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-16T11:17:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror_parent' already exists 2024-09-16T11:17:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-16T11:17:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-16T11:17:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:17:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-16T11:17:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-16T11:17:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-16T11:17:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-16T11:17:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-16T11:17:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z ']' 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z ']' 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_add cluster2 mirror cluster1 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local client_cluster=cluster1 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local remote_cluster=cluster1 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local uuid_var_name 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local error_code 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local peer_uuid 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:17:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_uuid=46b15b52-1b53-463c-9788-fe9521a08013 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 -eq 0 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:17:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror_parent 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local client_cluster=cluster1 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local remote_cluster=cluster1 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local uuid_var_name 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local error_code 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local peer_uuid 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:17:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:17:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ peer_uuid=17418ad6-adc7-4916-8b2a-6cbbe7aadaf1 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 -eq 0 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:17:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:17:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirrors cluster1 2024-09-16T11:17:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 0 3 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster1:0 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:0 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster1:0 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster1 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=0 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster1 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=0 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster1:1 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:1 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster1:1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster1:2 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:2 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster1:2 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster1 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=2 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster1 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=2 2024-09-16T11:17:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster1:3 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:3 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster1:3 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster1 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=3 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster1 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=3 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirrors cluster2 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 0 3 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster2:0 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:0 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster2:0 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster2 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=0 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster2 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster2 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=0 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster2:1 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:1 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster2:1 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster2 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=1 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster2 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster2 2024-09-16T11:17:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=1 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster2:2 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:2 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster2:2 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster2 2024-09-16T11:17:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=2 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster2 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster2 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=2 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster2:3 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:3 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster2:3 2024-09-16T11:17:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster2 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=3 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster2 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster2 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=3 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-16T11:17:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:17:12 TEST: add image and test replay after client crashes 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:17:12 TEST: add image and test replay after client crashes 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=test 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror test 512M 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:17:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:17:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:17:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2024-09-16T11:17:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:17:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-16T11:17:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:17:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-16T11:17:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:17:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-16T11:17:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:17:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:17:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10ad1403a957 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10ad1403a957 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:12 2024 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:17:12 2024 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:12 2024 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:17:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:17:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:17:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:17:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:17:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:17:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:17:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:17:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:17:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:17:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:17:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:17:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:17:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:17:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:17:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:17:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:17:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:17:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:17:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:17:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:17:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:17:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:17:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:17:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:17:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:17:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:17:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:17:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:17:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 1 2024-09-16T11:17:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:17:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:17:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:17:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:17:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:17:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:17:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:17:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:17:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:17:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:17:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:17:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:17:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:17:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:17:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:17:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:17:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:17:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:17:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:17:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:17:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:17:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 2 2024-09-16T11:17:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:17:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:17:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:17:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:17:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:17:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:17:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:17:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:17:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:17:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:17:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:17:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:17:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:17:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:17:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:17:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:17:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:17:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:17:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:17:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:17:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:17:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:17:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:17:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:17:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 4 2024-09-16T11:17:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:17:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:17:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:17:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:17:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:17:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:17:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:17:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:17:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:17:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:17:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:17:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:17:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:17:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:17:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:17:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:17:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:17:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:17:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:17:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 8 2024-09-16T11:17:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:17:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:17:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:17:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:17:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:17:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:17:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:17:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:17:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:17:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:17:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:17:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:17:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:17:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:17:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:17:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:17:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:17:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:17:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:17:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:17:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:17:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:17:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:17:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:17:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:17:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:17:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 8 2024-09-16T11:17:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:17:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:17:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:17:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:17:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:17:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:17:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:17:38.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:17:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:17:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:17:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:17:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:17:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:17:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:17:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:17:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:17:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:17:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:17:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:17:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:17:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:17:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:17:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:17:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:17:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:17:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:17:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:17:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:17:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:17:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:17:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:17:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name= 2024-09-16T11:17:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 10 2024-09-16T11:17:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:17:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:17:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:17:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:17:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:17:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:17:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:17:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 105224192 bytes 2024-09-16T11:17:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 149236736 bytes 2024-09-16T11:17:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 178244608 bytes 2024-09-16T11:17:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 223031296 bytes 2024-09-16T11:17:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 282128384 bytes 2024-09-16T11:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 301993984 bytes 2024-09-16T11:17:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 376597504 bytes 2024-09-16T11:17:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 461875200 bytes 2024-09-16T11:17:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 481596416 bytes 2024-09-16T11:17:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 502247424 bytes 2024-09-16T11:17:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 512770048 bytes 2024-09-16T11:17:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 535972864 bytes 2024-09-16T11:17:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 555811840 bytes 2024-09-16T11:17:40.817 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2685279681114068) > weight (0.25) 2024-09-16T11:17:40.818 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T11:17:40.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T11:17:40.819 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-16T11:17:40.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:17:40.817+0000 7f8aaa7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38257) UID: 1000 2024-09-16T11:17:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 599802880 bytes 2024-09-16T11:17:41.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:17:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 677819392 bytes 2024-09-16T11:17:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 716109824 bytes 2024-09-16T11:17:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 751891456 bytes 2024-09-16T11:17:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 771789824 bytes 2024-09-16T11:17:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 800470016 bytes 2024-09-16T11:17:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 875406336 bytes 2024-09-16T11:17:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 876256256 bytes 2024-09-16T11:17:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 925760512 bytes 2024-09-16T11:17:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 980578304 bytes 2024-09-16T11:17:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1026352128 bytes 2024-09-16T11:17:42.820 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T11:17:42.820 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:17:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:17:42.820 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T11:17:42.820 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T11:17:42.821 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T11:17:42.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T11:17:42.824 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-09-16T11:17:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1077776384 bytes 2024-09-16T11:17:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1125151744 bytes 2024-09-16T11:17:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1157807104 bytes 2024-09-16T11:17:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1184488448 bytes 2024-09-16T11:17:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1233492992 bytes 2024-09-16T11:17:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1306672128 bytes 2024-09-16T11:17:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1349061632 bytes 2024-09-16T11:17:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1393060864 bytes 2024-09-16T11:17:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1448863744 bytes 2024-09-16T11:17:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1509045248 bytes 2024-09-16T11:17:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1543163904 bytes 2024-09-16T11:17:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1543783424 bytes 2024-09-16T11:17:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1567583232 bytes 2024-09-16T11:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1588941824 bytes 2024-09-16T11:17:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1632565248 bytes 2024-09-16T11:17:44.818 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6838530321072995) > weight (0.25) 2024-09-16T11:17:44.818 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8571378973523829) > weight (0.25) 2024-09-16T11:17:44.818 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T11:17:44.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T11:17:44.818 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T11:17:44.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:17:44.818+0000 7f8e6edf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38159) UID: 1000 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:17:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:17:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:17:45.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+stopped 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: stopped 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:17:44 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: ' 2024-09-16T11:17:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:17:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 1 2024-09-16T11:17:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 2 2024-09-16T11:17:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:17:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:17:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:17:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:17:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+stopped 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: stopped 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:17:44 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: ' 2024-09-16T11:17:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:17:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 1 2024-09-16T11:17:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 4 2024-09-16T11:17:48.819 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T11:17:48.819 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:17:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:17:48.819 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T11:17:48.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T11:17:48.820 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T11:17:48.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T11:17:48.822 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-16T11:17:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:17:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:17:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:17:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: ' 2024-09-16T11:17:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:17:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 1 2024-09-16T11:17:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:17:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 8 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:17:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:17:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: ' 2024-09-16T11:18:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:18:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 1 2024-09-16T11:18:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:18:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 8 2024-09-16T11:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:18:03' 2024-09-16T11:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 1 2024-09-16T11:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 8 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:18:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:18:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:18:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:18:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: down+unknown 2024-09-16T11:18:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: status not found 2024-09-16T11:18:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16T11:18:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:18:03' 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 1 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:18:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 8 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:18:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:18:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:18:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:18:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:18:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485432,"remote_snapshot_timestamp":1726485432,"replay_state":"idle"} 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:18:17 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:18:03' 2024-09-16T11:18:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:18:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:18:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:18:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485432,"remote_snapshot_timestamp":1726485432,"replay_state":"idle"} 2024-09-16T11:18:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:18:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:18:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:18:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:18:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:18:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:18:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:18:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:18:17' 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:18:17' 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:18:17' 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:18:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42144294mirror.1smithi10317.2.7-1643-gd75f2382falseOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485432,"remote_snapshot_timestamp":1726485432,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:18:17cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:18:03' 2024-09-16T11:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:18:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:18:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:18:17' 2024-09-16T11:18:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:18:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:18:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:18:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:18:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:18:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap1 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap1 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1 2024-09-16T11:18:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:18:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:18:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:18:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:18:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:18:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:18:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:18:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:18:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:18:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:18:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:18:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:18:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:18:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:18:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:18:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:18:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:18:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:18:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:18:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:18:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:18:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:18:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:18:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:18:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:18:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:18:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:18:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:18:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:18:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:18:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:18:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:18:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:18:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:18:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:18:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:18:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:18:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:18:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:18:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:18:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:18:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:18:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:18:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:18:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:18:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:18:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:18:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:18:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:18:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:18:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:18:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:18:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:18:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:18:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:18:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:18:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:18:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:18:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:18:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:18:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:18:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 6 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:18:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:18:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:18:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=6 2024-09-16T11:18:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:18:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:18:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:18:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:18:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:18:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=4 2024-09-16T11:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 6 = 4 2024-09-16T11:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:18:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:18:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:18:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:18:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:18:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:18:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:18:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:18:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:18:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:18:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 6 = '' 2024-09-16T11:18:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:18:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:18:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:18:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:18:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:18:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:18:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:18:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:18:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:18:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:18:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 6 = '' 2024-09-16T11:18:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:18:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:18:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:18:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=6 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 6 = 6 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1 2024-09-16T11:18:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:18:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:18:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:18:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-16T11:18:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:18:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:18:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:18:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:18:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:18:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:18:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap1 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:18:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:18:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:18:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:18:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:18:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:18:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:18:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap1-clean 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1-clean 2024-09-16T11:18:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:18:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:18:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:18:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:18:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:18:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:18:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:18:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 29099008 bytes 2024-09-16T11:18:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 95192064 bytes 2024-09-16T11:18:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 124242944 bytes 2024-09-16T11:18:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 229743616 bytes 2024-09-16T11:18:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 308256768 bytes 2024-09-16T11:18:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 373168128 bytes 2024-09-16T11:18:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 413928448 bytes 2024-09-16T11:18:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 434449408 bytes 2024-09-16T11:18:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 463792128 bytes 2024-09-16T11:18:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 510789632 bytes 2024-09-16T11:18:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 563934208 bytes 2024-09-16T11:18:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 625875968 bytes 2024-09-16T11:18:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 631234560 bytes 2024-09-16T11:18:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 680550400 bytes 2024-09-16T11:18:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 773279744 bytes 2024-09-16T11:18:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 800976896 bytes 2024-09-16T11:18:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 822629376 bytes 2024-09-16T11:18:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 826463232 bytes 2024-09-16T11:18:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 910630912 bytes 2024-09-16T11:18:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 969112576 bytes 2024-09-16T11:18:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1046774784 bytes 2024-09-16T11:18:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1130379264 bytes 2024-09-16T11:18:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1200958464 bytes 2024-09-16T11:18:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1223639040 bytes 2024-09-16T11:18:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1271437312 bytes 2024-09-16T11:18:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1299010560 bytes 2024-09-16T11:18:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1324358656 bytes 2024-09-16T11:18:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1363810304 bytes 2024-09-16T11:18:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1369844736 bytes 2024-09-16T11:18:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1389634560 bytes 2024-09-16T11:18:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1487931392 bytes 2024-09-16T11:18:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1561490432 bytes 2024-09-16T11:18:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1583932416 bytes 2024-09-16T11:18:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1655214080 bytes 2024-09-16T11:18:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1698553856 bytes 2024-09-16T11:18:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1780582400 bytes 2024-09-16T11:18:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1848349696 bytes 2024-09-16T11:18:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1961750528 bytes 2024-09-16T11:18:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 2021487616 bytes 2024-09-16T11:18:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 2022099968 bytes 2024-09-16T11:18:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 2030616576 bytes 2024-09-16T11:18:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 2049040384 bytes 2024-09-16T11:18:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2124893184 bytes 2024-09-16T11:18:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2147838976 bytes 2024-09-16T11:18:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2250386432 bytes 2024-09-16T11:18:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2270622720 bytes 2024-09-16T11:18:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2303349760 bytes 2024-09-16T11:18:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2402912256 bytes 2024-09-16T11:18:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2426973184 bytes 2024-09-16T11:18:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2467276800 bytes 2024-09-16T11:18:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2540144640 bytes 2024-09-16T11:18:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2547415040 bytes 2024-09-16T11:18:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2572018688 bytes 2024-09-16T11:18:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2632322048 bytes 2024-09-16T11:18:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:18:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:18:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:18:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:18:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:18:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:18:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:18:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":31110272.0,"bytes_per_snapshot":497764352.0,"last_snapshot_bytes":497764352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485507,"remote_snapshot_timestamp":1726485507,"replay_state":"idle"} 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:18:34 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:18:33' 2024-09-16T11:18:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:18:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:18:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:18:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":31110272.0,"bytes_per_snapshot":497764352.0,"last_snapshot_bytes":497764352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485507,"remote_snapshot_timestamp":1726485507,"replay_state":"idle"} 2024-09-16T11:18:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:18:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:18:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:18:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:18:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:18:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:18:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:18:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:18:34' 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:18:34' 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:18:34' 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:18:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:18:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42144294mirror.1smithi10317.2.7-1643-gd75f2382falseOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":31110272.0,"bytes_per_snapshot":497764352.0,"last_snapshot_bytes":497764352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485507,"remote_snapshot_timestamp":1726485507,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:18:34cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:18:33' 2024-09-16T11:18:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:18:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:18:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:18:34' 2024-09-16T11:18:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:18:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:18:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:18:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:18:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap2 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap2 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2 2024-09-16T11:18:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:18:59.823 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8355389406752963) > weight (0.25) 2024-09-16T11:18:59.823 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T11:18:59.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T11:18:59.824 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T11:18:59.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:18:59.823+0000 7f90839f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 38142) UID: 1000 2024-09-16T11:19:00.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:19:00.825 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T11:19:00.825 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:19:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:19:00.826 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T11:19:00.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:19:00.826 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:19:00.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:19:00.829 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:19:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:19:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:19:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:19:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:19:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:19:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:19:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:19:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:19:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:19:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:19:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:19:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:19:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:19:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:19:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:19:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:19:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:19:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:19:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:19:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:19:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:19:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:19:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:19:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:19:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:19:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:19:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:19:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:19:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:19:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:19:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:19:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:19:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:19:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:19:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:19:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:19:13.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:19:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:19:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:19:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:19:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:19:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:19:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:19:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:19:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:19:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:19:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:19:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:19:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:19:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:19:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:19:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:19:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:19:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:19:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:19:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:19:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:19:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:19:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:19:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:19:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:19:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:19:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:19:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:19:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 9 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:19:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=9 2024-09-16T11:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:19:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:19:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:19:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:19:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:19:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:19:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:19:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:19:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:19:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 9 = '' 2024-09-16T11:19:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:19:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:19:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:19:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:19:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 9 = '' 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:19:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:19:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:19:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:19:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:19:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:19:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:19:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:19:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:19:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 9 = '' 2024-09-16T11:19:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:19:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:19:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:19:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:19:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:19:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=9 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 9 = 9 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2 2024-09-16T11:19:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:19:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-16T11:19:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:19:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:19:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:19:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 4 2024-09-16T11:19:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:19:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:19:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:19:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:19:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:19:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:19:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:19:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:19:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:19:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:19:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap1-clean ']' 2024-09-16T11:19:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-16T11:19:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1-clean 2024-09-16T11:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:19:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:19:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:19:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap2 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:19:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:19:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:19:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap2-clean 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2-clean 2024-09-16T11:19:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:19:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:19:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:19:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:19:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=13 2024-09-16T11:19:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:19:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:19:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 19096576 bytes 2024-09-16T11:19:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 94609408 bytes 2024-09-16T11:19:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 135025664 bytes 2024-09-16T11:19:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 190477312 bytes 2024-09-16T11:19:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 221146112 bytes 2024-09-16T11:19:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 302384128 bytes 2024-09-16T11:19:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 309853184 bytes 2024-09-16T11:19:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 318321664 bytes 2024-09-16T11:19:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 355426304 bytes 2024-09-16T11:19:24.825 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2691387743705871) > weight (0.25) 2024-09-16T11:19:24.825 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T11:19:24.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T11:19:24.826 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T11:19:24.826 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T11:19:24.826 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:19:24.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:19:24.825+0000 7fa9d47f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39814) UID: 1000 2024-09-16T11:19:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 399557632 bytes 2024-09-16T11:19:25.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:19:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 419786752 bytes 2024-09-16T11:19:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 459953152 bytes 2024-09-16T11:19:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 507748352 bytes 2024-09-16T11:19:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 560374784 bytes 2024-09-16T11:19:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 578267136 bytes 2024-09-16T11:19:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 589168640 bytes 2024-09-16T11:19:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 648407040 bytes 2024-09-16T11:19:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 673220608 bytes 2024-09-16T11:19:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 702419968 bytes 2024-09-16T11:19:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 718220288 bytes 2024-09-16T11:19:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 745555968 bytes 2024-09-16T11:19:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 820569088 bytes 2024-09-16T11:19:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 851415040 bytes 2024-09-16T11:19:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 918028288 bytes 2024-09-16T11:19:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 931806208 bytes 2024-09-16T11:19:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 991755264 bytes 2024-09-16T11:19:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1060954112 bytes 2024-09-16T11:19:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1164737536 bytes 2024-09-16T11:19:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1185957888 bytes 2024-09-16T11:19:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1235525632 bytes 2024-09-16T11:19:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1282293760 bytes 2024-09-16T11:19:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1327599616 bytes 2024-09-16T11:19:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1380737024 bytes 2024-09-16T11:19:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1454137344 bytes 2024-09-16T11:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1513937920 bytes 2024-09-16T11:19:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1537399808 bytes 2024-09-16T11:19:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1550898176 bytes 2024-09-16T11:19:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1640754176 bytes 2024-09-16T11:19:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1716556800 bytes 2024-09-16T11:19:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1742053376 bytes 2024-09-16T11:19:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1794989056 bytes 2024-09-16T11:19:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1871762432 bytes 2024-09-16T11:19:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1889003520 bytes 2024-09-16T11:19:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1913219072 bytes 2024-09-16T11:19:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1976110080 bytes 2024-09-16T11:19:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2046183424 bytes 2024-09-16T11:19:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2089191424 bytes 2024-09-16T11:19:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2130078720 bytes 2024-09-16T11:19:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2151633920 bytes 2024-09-16T11:19:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2165880832 bytes 2024-09-16T11:19:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:19:30.927 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T11:19:30.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T11:19:30.928 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T11:19:30.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T11:19:30.933 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-16T11:19:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2250712064 bytes 2024-09-16T11:19:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2329226240 bytes 2024-09-16T11:19:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2329834496 bytes 2024-09-16T11:19:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2373640192 bytes 2024-09-16T11:19:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2434453504 bytes 2024-09-16T11:19:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2515702784 bytes 2024-09-16T11:19:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2577931264 bytes 2024-09-16T11:19:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2648883200 bytes 2024-09-16T11:19:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2711383040 bytes 2024-09-16T11:19:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2796203008 bytes 2024-09-16T11:19:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2821075968 bytes 2024-09-16T11:19:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2847882240 bytes 2024-09-16T11:19:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2914065408 bytes 2024-09-16T11:19:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2997248000 bytes 2024-09-16T11:19:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 3046554624 bytes 2024-09-16T11:19:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 3077521408 bytes 2024-09-16T11:19:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 3104397312 bytes 2024-09-16T11:19:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 3132426240 bytes 2024-09-16T11:19:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 3133022208 bytes 2024-09-16T11:19:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3136215040 bytes 2024-09-16T11:19:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3165739008 bytes 2024-09-16T11:19:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3233539072 bytes 2024-09-16T11:19:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3254426624 bytes 2024-09-16T11:19:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3329598464 bytes 2024-09-16T11:19:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3387213824 bytes 2024-09-16T11:19:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3459759104 bytes 2024-09-16T11:19:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3517679616 bytes 2024-09-16T11:19:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3582638080 bytes 2024-09-16T11:19:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3630682112 bytes 2024-09-16T11:19:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3702145024 bytes 2024-09-16T11:19:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3757679616 bytes 2024-09-16T11:19:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3828038656 bytes 2024-09-16T11:19:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3876116480 bytes 2024-09-16T11:19:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3892579328 bytes 2024-09-16T11:19:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:19:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:19:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:19:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:19:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:19:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:19:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:19:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:19:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:19:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:19:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:19:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:19:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":17772987.73,"bytes_per_snapshot":515476992.0,"last_snapshot_bytes":533189632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485553,"remote_snapshot_timestamp":1726485553,"replay_state":"idle"} 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:19:34 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:19:33' 2024-09-16T11:19:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:19:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:19:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:19:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":17772987.73,"bytes_per_snapshot":515476992.0,"last_snapshot_bytes":533189632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485553,"remote_snapshot_timestamp":1726485553,"replay_state":"idle"} 2024-09-16T11:19:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:19:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:19:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:19:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:19:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:19:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:19:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:19:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:19:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:19:34' 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:19:34' 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:19:34' 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:19:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:19:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOK43744377mirror.1smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":17772987.73,"bytes_per_snapshot":515476992.0,"last_snapshot_bytes":533189632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485553,"remote_snapshot_timestamp":1726485553,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:19:34cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:19:33' 2024-09-16T11:19:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:19:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:19:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:19:34' 2024-09-16T11:19:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:19:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:19:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:19:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:19:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:19:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap3 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap3 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3 2024-09-16T11:19:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:20:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:20:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:20:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:20:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:20:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:20:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:20:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:20:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:20:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:20:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:20:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:20:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:20:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:20:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:20:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:20:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:20:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:20:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:20:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:20:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:20:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:20:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:20:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:20:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:20:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:20:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:20:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:20:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:20:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:20:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:20:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:20:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:20:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:20:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:20:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:20:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:20:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:20:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:20:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:20:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:20:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:20:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:20:08.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:20:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:20:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:20:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:20:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:20:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:20:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:20:08.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.33115539029272656) > weight (0.25) 2024-09-16T11:20:08.830 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T11:20:08.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T11:20:08.830 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T11:20:08.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:20:08.830+0000 7f13587f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 40590) UID: 1000 2024-09-16T11:20:09.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:20:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 13 2024-09-16T11:20:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:20:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=13 2024-09-16T11:20:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:20:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 13 = '' 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:20:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:20:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:20:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:20:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:20:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 13 = '' 2024-09-16T11:20:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:20:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:20:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:20:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:20:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 13 = '' 2024-09-16T11:20:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:20:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=13 2024-09-16T11:20:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 13 = 13 2024-09-16T11:20:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-16T11:20:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3 2024-09-16T11:20:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:20:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:20:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:20:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 6 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:20:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:19:19 2024 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap2-clean ']' 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2-clean 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:20:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap3 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:16.832 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T11:20:16.832 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:20:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:20:16.832 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T11:20:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:20:16.832 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:20:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:20:16.838 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-16T11:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap3-clean 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3-clean 2024-09-16T11:20:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:20:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:20:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:20:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:20:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:20:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:20:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 27064320 bytes 2024-09-16T11:20:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 77471744 bytes 2024-09-16T11:20:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 148302848 bytes 2024-09-16T11:20:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 181855232 bytes 2024-09-16T11:20:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 207509504 bytes 2024-09-16T11:20:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 243499008 bytes 2024-09-16T11:20:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 246736896 bytes 2024-09-16T11:20:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 283379712 bytes 2024-09-16T11:20:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 309387264 bytes 2024-09-16T11:20:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 387084288 bytes 2024-09-16T11:20:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 472799232 bytes 2024-09-16T11:20:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 525383680 bytes 2024-09-16T11:20:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 553776128 bytes 2024-09-16T11:20:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 583140352 bytes 2024-09-16T11:20:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 583728128 bytes 2024-09-16T11:20:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 645326848 bytes 2024-09-16T11:20:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 696496128 bytes 2024-09-16T11:20:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 697118720 bytes 2024-09-16T11:20:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 767471616 bytes 2024-09-16T11:20:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 790106112 bytes 2024-09-16T11:20:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 807927808 bytes 2024-09-16T11:20:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 821202944 bytes 2024-09-16T11:20:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 890133504 bytes 2024-09-16T11:20:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 942870528 bytes 2024-09-16T11:20:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 962601984 bytes 2024-09-16T11:20:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 991676416 bytes 2024-09-16T11:20:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1061507072 bytes 2024-09-16T11:20:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1085949952 bytes 2024-09-16T11:20:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1110269952 bytes 2024-09-16T11:20:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1144069120 bytes 2024-09-16T11:20:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1211589632 bytes 2024-09-16T11:20:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1250685952 bytes 2024-09-16T11:20:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1328048128 bytes 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:20:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:20:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:20:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:20:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:20:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:20:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:20:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:20:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:20:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:20:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:20:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:20:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:20:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:20:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":515476992.0,"last_snapshot_bytes":533189632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485553,"remote_snapshot_timestamp":1726485553,"replay_state":"idle"} 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:20:04 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:20:03' 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:20:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:20:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:20:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":515476992.0,"last_snapshot_bytes":533189632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485553,"remote_snapshot_timestamp":1726485553,"replay_state":"idle"} 2024-09-16T11:20:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:20:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:20:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:20:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:20:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:20:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:20:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:20:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:20:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:20:04' 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:20:04' 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:20:04' 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:20:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:20:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOK44384441mirror.1smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":515476992.0,"last_snapshot_bytes":533189632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485553,"remote_snapshot_timestamp":1726485553,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:20:04cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:20:03' 2024-09-16T11:20:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:20:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:20:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:20:04' 2024-09-16T11:20:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:20:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:20:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:20:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:20:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:20:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap4 2024-09-16T11:20:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap4 2024-09-16T11:20:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4 2024-09-16T11:20:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:20:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:20:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:20:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:20:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:20:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:20:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:20:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:20:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:20:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:20:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:20:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:20:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:20:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:20:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:20:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:20:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:20:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:20:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:20:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:20:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:20:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:20:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:20:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:20:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:20:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:20:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:20:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:20:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:20:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:20:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:20:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:20:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:20:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:20:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:20:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:20:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:20:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:20:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:20:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:20:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:20:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:20:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:20:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:20:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:20:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:20:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:20:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:20:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:20:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:20:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:20:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:20:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:20:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:20:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:20:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 17 2024-09-16T11:20:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:20:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:20:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:20:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:20:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:20:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=17 2024-09-16T11:20:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:20:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:20:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:20:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:20:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:20:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 17 = '' 2024-09-16T11:20:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:20:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:20:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:20:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:20:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:20:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 17 = '' 2024-09-16T11:20:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:20:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:20:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:20:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:20:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:20:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:20:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 17 = '' 2024-09-16T11:20:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:20:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:20:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:20:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=17 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 17 = 17 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4 2024-09-16T11:20:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:20:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:20:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:20:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-16T11:20:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:20:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:20:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 8 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:19:19 2024 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:20:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:20:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:20:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap3-clean ']' 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3-clean 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:20:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap4 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:20:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:20:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:20:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:21:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap4-clean 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4-clean 2024-09-16T11:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:21:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:21:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:21:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:21:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:21:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:21:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:21:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 15703040 bytes 2024-09-16T11:21:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 46796800 bytes 2024-09-16T11:21:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 103102464 bytes 2024-09-16T11:21:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 118900736 bytes 2024-09-16T11:21:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 183859200 bytes 2024-09-16T11:21:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 213122048 bytes 2024-09-16T11:21:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 244915200 bytes 2024-09-16T11:21:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 312310784 bytes 2024-09-16T11:21:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 335476736 bytes 2024-09-16T11:21:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 388407296 bytes 2024-09-16T11:21:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 430852096 bytes 2024-09-16T11:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 589604864 bytes 2024-09-16T11:21:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 667255808 bytes 2024-09-16T11:21:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 702194688 bytes 2024-09-16T11:21:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 749630464 bytes 2024-09-16T11:21:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 788872192 bytes 2024-09-16T11:21:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 820763648 bytes 2024-09-16T11:21:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 826761216 bytes 2024-09-16T11:21:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 849390592 bytes 2024-09-16T11:21:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 902218752 bytes 2024-09-16T11:21:04.933 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.942809554619285) > weight (0.25) 2024-09-16T11:21:04.934 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7962987101735639) > weight (0.25) 2024-09-16T11:21:04.934 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T11:21:04.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T11:21:04.934 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T11:21:04.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:21:04.933+0000 7f52133f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 38305) UID: 1000 2024-09-16T11:21:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 956576768 bytes 2024-09-16T11:21:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 991557632 bytes 2024-09-16T11:21:05.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:21:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1054040064 bytes 2024-09-16T11:21:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1106820096 bytes 2024-09-16T11:21:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1161373696 bytes 2024-09-16T11:21:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1161987072 bytes 2024-09-16T11:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1261732864 bytes 2024-09-16T11:21:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1307076608 bytes 2024-09-16T11:21:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1342912512 bytes 2024-09-16T11:21:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1363438592 bytes 2024-09-16T11:21:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1379013632 bytes 2024-09-16T11:21:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1446705152 bytes 2024-09-16T11:21:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1447292928 bytes 2024-09-16T11:21:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1474846720 bytes 2024-09-16T11:21:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1563134976 bytes 2024-09-16T11:21:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1597181952 bytes 2024-09-16T11:21:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1651389440 bytes 2024-09-16T11:21:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1694007296 bytes 2024-09-16T11:21:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1740979200 bytes 2024-09-16T11:21:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1764706304 bytes 2024-09-16T11:21:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1818461184 bytes 2024-09-16T11:21:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1854973952 bytes 2024-09-16T11:21:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1878606848 bytes 2024-09-16T11:21:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1945315328 bytes 2024-09-16T11:21:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1980482560 bytes 2024-09-16T11:21:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2028507136 bytes 2024-09-16T11:21:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2092894208 bytes 2024-09-16T11:21:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2143286272 bytes 2024-09-16T11:21:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2185465856 bytes 2024-09-16T11:21:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2246591488 bytes 2024-09-16T11:21:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2282542080 bytes 2024-09-16T11:21:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2324557824 bytes 2024-09-16T11:21:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2355367936 bytes 2024-09-16T11:21:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2419870720 bytes 2024-09-16T11:21:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2445365248 bytes 2024-09-16T11:21:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2493224960 bytes 2024-09-16T11:21:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2586289152 bytes 2024-09-16T11:21:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2647901184 bytes 2024-09-16T11:21:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2691860480 bytes 2024-09-16T11:21:10.935 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T11:21:10.935 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:21:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:21:10.936 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T11:21:10.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T11:21:10.936 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T11:21:10.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T11:21:10.939 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-16T11:21:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2737002496 bytes 2024-09-16T11:21:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2770490368 bytes 2024-09-16T11:21:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2827422720 bytes 2024-09-16T11:21:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2887382016 bytes 2024-09-16T11:21:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2927060992 bytes 2024-09-16T11:21:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 3070158848 bytes 2024-09-16T11:21:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 3073193984 bytes 2024-09-16T11:21:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 3085307904 bytes 2024-09-16T11:21:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 3163228160 bytes 2024-09-16T11:21:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 3220703232 bytes 2024-09-16T11:21:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3280415744 bytes 2024-09-16T11:21:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3326262272 bytes 2024-09-16T11:21:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3334360064 bytes 2024-09-16T11:21:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3337544704 bytes 2024-09-16T11:21:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3374651392 bytes 2024-09-16T11:21:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3412037632 bytes 2024-09-16T11:21:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3442562048 bytes 2024-09-16T11:21:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3530880000 bytes 2024-09-16T11:21:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3573063680 bytes 2024-09-16T11:21:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3625720832 bytes 2024-09-16T11:21:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3649641472 bytes 2024-09-16T11:21:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3672267776 bytes 2024-09-16T11:21:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3692529664 bytes 2024-09-16T11:21:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3707427840 bytes 2024-09-16T11:21:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3737965568 bytes 2024-09-16T11:21:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3793411072 bytes 2024-09-16T11:21:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8600 IOs, 3840883712 bytes 2024-09-16T11:21:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8700 IOs, 3906861056 bytes 2024-09-16T11:21:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8800 IOs, 3940167680 bytes 2024-09-16T11:21:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8900 IOs, 3965535232 bytes 2024-09-16T11:21:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9000 IOs, 3966142464 bytes 2024-09-16T11:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9100 IOs, 3970360320 bytes 2024-09-16T11:21:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:21:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:21:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:21:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:21:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:21:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":16645461.33,"bytes_per_snapshot":517340160.0,"last_snapshot_bytes":499363840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485651,"remote_snapshot_timestamp":1726485651,"replay_state":"idle"} 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:21:04 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:21:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:21:03' 2024-09-16T11:21:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:21:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:21:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:21:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":16645461.33,"bytes_per_snapshot":517340160.0,"last_snapshot_bytes":499363840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485651,"remote_snapshot_timestamp":1726485651,"replay_state":"idle"} 2024-09-16T11:21:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:21:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:21:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:21:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:21:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:21:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:21:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:21:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:21:04' 2024-09-16T11:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:21:04' 2024-09-16T11:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:21:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:21:04' 2024-09-16T11:21:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:21:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:21:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:21:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOK44384441mirror.1smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":16645461.33,"bytes_per_snapshot":517340160.0,"last_snapshot_bytes":499363840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485651,"remote_snapshot_timestamp":1726485651,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:21:04cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:21:03' 2024-09-16T11:21:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:21:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:21:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:21:04' 2024-09-16T11:21:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:21:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:21:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:21:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:21:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:21:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap5 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap5 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5 2024-09-16T11:21:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:21:37.840 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5755877506161089) > weight (0.25) 2024-09-16T11:21:37.840 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8391897815106526) > weight (0.25) 2024-09-16T11:21:37.840 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5331210363464631) > weight (0.25) 2024-09-16T11:21:37.840 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8988074869928793) > weight (0.25) 2024-09-16T11:21:37.840 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T11:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:21:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:21:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:21:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:21:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:21:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:21:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:21:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:21:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:21:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:21:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:21:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:21:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:21:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:21:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:21:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:21:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:21:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:21:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:21:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:21:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:21:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:21:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:21:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:21:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:21:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:21:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:21:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:21:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:21:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:21:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:21:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:21:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:21:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:21:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:21:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:21:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:21:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:21:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:21:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:21:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:21:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:21:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:21:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:21:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:21:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:21:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:21:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:21:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:21:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:21:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:21:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:21:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:21:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:21:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:21:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:21:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:21:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:21:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:21:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:21:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:21:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:21:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:21:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:21:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:21:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:21:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:21:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:21:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:21:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:21:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:21:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:21:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:21:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:21:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 21 2024-09-16T11:21:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:21:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:21:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:21:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:21:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:21:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=21 2024-09-16T11:21:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:21:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:21:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:21:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:21:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:21:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:21:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:21:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:21:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:21:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:21:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 21 = '' 2024-09-16T11:21:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:21:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:21:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:21:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:21:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:21:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:21:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:21:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:21:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 21 = '' 2024-09-16T11:21:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:21:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:21:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:21:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 21 = '' 2024-09-16T11:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:21:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:21:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:21:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:21:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=21 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 21 = 21 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:21:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:21:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 10 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:21:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:20:57 2024 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:21:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap4-clean ']' 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4-clean 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:21:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:21:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:21:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap5 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:21:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:21:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:21:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:21:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap5-clean 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5-clean 2024-09-16T11:21:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:21:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:21:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:21:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:21:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:21:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:21:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:21:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=11 2024-09-16T11:21:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:21:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:21:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 29448192 bytes 2024-09-16T11:21:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 82310144 bytes 2024-09-16T11:21:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 93526016 bytes 2024-09-16T11:21:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 180655104 bytes 2024-09-16T11:21:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 188170240 bytes 2024-09-16T11:21:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 214670336 bytes 2024-09-16T11:21:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 243741696 bytes 2024-09-16T11:21:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 306956288 bytes 2024-09-16T11:21:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 327132160 bytes 2024-09-16T11:21:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 334305280 bytes 2024-09-16T11:21:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 334923776 bytes 2024-09-16T11:21:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 351595520 bytes 2024-09-16T11:21:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 384257024 bytes 2024-09-16T11:21:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 426169344 bytes 2024-09-16T11:21:59.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 492360704 bytes 2024-09-16T11:21:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 542401536 bytes 2024-09-16T11:21:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 597472256 bytes 2024-09-16T11:22:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 657814528 bytes 2024-09-16T11:22:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 702209024 bytes 2024-09-16T11:22:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 750187520 bytes 2024-09-16T11:22:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 757634048 bytes 2024-09-16T11:22:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 818785280 bytes 2024-09-16T11:22:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 874998784 bytes 2024-09-16T11:22:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 970676224 bytes 2024-09-16T11:22:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 977860608 bytes 2024-09-16T11:22:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1064586240 bytes 2024-09-16T11:22:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1105855488 bytes 2024-09-16T11:22:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1149016064 bytes 2024-09-16T11:22:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1222333440 bytes 2024-09-16T11:22:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1229734912 bytes 2024-09-16T11:22:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1297090560 bytes 2024-09-16T11:22:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1327632384 bytes 2024-09-16T11:22:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1376261120 bytes 2024-09-16T11:22:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1404379136 bytes 2024-09-16T11:22:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1482771456 bytes 2024-09-16T11:22:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1513336832 bytes 2024-09-16T11:22:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1527674880 bytes 2024-09-16T11:22:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1560099840 bytes 2024-09-16T11:22:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1560722432 bytes 2024-09-16T11:22:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1617732608 bytes 2024-09-16T11:22:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1703037952 bytes 2024-09-16T11:22:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1784170496 bytes 2024-09-16T11:22:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1818596352 bytes 2024-09-16T11:22:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1835756544 bytes 2024-09-16T11:22:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1914816512 bytes 2024-09-16T11:22:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1937690624 bytes 2024-09-16T11:22:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2008799232 bytes 2024-09-16T11:22:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2010450944 bytes 2024-09-16T11:22:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2026157056 bytes 2024-09-16T11:22:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2056477696 bytes 2024-09-16T11:22:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2071491584 bytes 2024-09-16T11:22:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2135597056 bytes 2024-09-16T11:22:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2157062144 bytes 2024-09-16T11:22:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2176453632 bytes 2024-09-16T11:22:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2261460992 bytes 2024-09-16T11:22:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2270840832 bytes 2024-09-16T11:22:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2289662976 bytes 2024-09-16T11:22:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2358363136 bytes 2024-09-16T11:22:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2408588288 bytes 2024-09-16T11:22:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2469663744 bytes 2024-09-16T11:22:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2485990400 bytes 2024-09-16T11:22:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2497411072 bytes 2024-09-16T11:22:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2542355456 bytes 2024-09-16T11:22:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2615010304 bytes 2024-09-16T11:22:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2650476544 bytes 2024-09-16T11:22:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2740719616 bytes 2024-09-16T11:22:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2791600128 bytes 2024-09-16T11:22:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2830257152 bytes 2024-09-16T11:22:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2891893760 bytes 2024-09-16T11:22:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2925629440 bytes 2024-09-16T11:22:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2960184320 bytes 2024-09-16T11:22:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2977170432 bytes 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:22:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:22:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:22:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:22:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:22:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":17838353.07,"bytes_per_snapshot":517257216.0,"last_snapshot_bytes":535150592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485707,"remote_snapshot_timestamp":1726485707,"replay_state":"idle"} 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:22:04 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:22:03' 2024-09-16T11:22:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:22:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:22:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:22:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":17838353.07,"bytes_per_snapshot":517257216.0,"last_snapshot_bytes":535150592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485707,"remote_snapshot_timestamp":1726485707,"replay_state":"idle"} 2024-09-16T11:22:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:22:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:22:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:22:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:22:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:22:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:22:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:22:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:22:04' 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:22:04' 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:22:04' 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:22:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOK44384441mirror.1smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":17838353.07,"bytes_per_snapshot":517257216.0,"last_snapshot_bytes":535150592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485707,"remote_snapshot_timestamp":1726485707,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:22:04cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:22:03' 2024-09-16T11:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:22:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:22:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:22:04' 2024-09-16T11:22:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:22:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:22:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:22:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap6 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap6 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6 2024-09-16T11:22:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:22:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:22:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:22:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:22:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:22:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:22:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:22:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:22:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:22:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:22:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:22:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:22:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:22:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:22:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:22:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:22:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:22:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:22:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:22:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:22:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:22:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:22:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:22:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:22:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:22:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:22:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:22:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:22:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:22:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:22:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:22:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:22:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:22:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:22:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:22:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:22:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:22:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:22:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:22:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:22:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:22:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:22:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:22:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:22:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:22:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:22:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:22:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:22:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:22:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:22:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:22:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:22:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:22:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:22:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:22:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:22:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:22:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:22:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:22:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:22:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:22:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:22:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:22:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:22:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:22:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:22:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:22:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:22:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:22:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:22:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:22:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 25 2024-09-16T11:22:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:22:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:22:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:22:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:22:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:22:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=25 2024-09-16T11:22:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:22:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:22:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:22:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:22:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:22:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:22:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:22:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:22:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:22:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:22:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 25 = '' 2024-09-16T11:22:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:22:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:22:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:22:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:22:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:22:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:22:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:22:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:22:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:22:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:22:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 25 = '' 2024-09-16T11:22:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:22:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:22:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:22:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:22:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:22:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:22:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:22:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:22:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:22:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 25 = '' 2024-09-16T11:22:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:22:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:22:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:22:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:22:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:22:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=25 2024-09-16T11:22:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 25 = 25 2024-09-16T11:22:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:22:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:22:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-16T11:22:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:22:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:22:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 12 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:20:57 2024 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:22:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:22:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap5-clean ']' 2024-09-16T11:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-16T11:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5-clean 2024-09-16T11:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:22:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:22:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:22:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap6 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:22:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:22:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:22:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:22:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:22:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap6-clean 2024-09-16T11:22:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-16T11:22:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6-clean 2024-09-16T11:22:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:22:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:22:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:22:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:22:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:22:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:22:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:22:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=13 2024-09-16T11:22:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:22:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:22:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 38353920 bytes 2024-09-16T11:22:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 47494144 bytes 2024-09-16T11:22:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 109330432 bytes 2024-09-16T11:22:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 148315136 bytes 2024-09-16T11:22:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 228896768 bytes 2024-09-16T11:22:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 302049280 bytes 2024-09-16T11:22:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 388263936 bytes 2024-09-16T11:22:51.939 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2790223307353764) > weight (0.25) 2024-09-16T11:22:51.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.44254674772406655) > weight (0.25) 2024-09-16T11:22:51.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6704950580423849) > weight (0.25) 2024-09-16T11:22:51.940 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5621721924649284) > weight (0.25) 2024-09-16T11:22:51.940 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-16T11:22:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 446764032 bytes 2024-09-16T11:22:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 540800000 bytes 2024-09-16T11:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 580559872 bytes 2024-09-16T11:22:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 601511936 bytes 2024-09-16T11:22:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 635996160 bytes 2024-09-16T11:22:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 679065600 bytes 2024-09-16T11:22:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 717704192 bytes 2024-09-16T11:22:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 732417024 bytes 2024-09-16T11:22:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 770083840 bytes 2024-09-16T11:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 825523200 bytes 2024-09-16T11:22:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 836099072 bytes 2024-09-16T11:22:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 852976640 bytes 2024-09-16T11:22:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 865287168 bytes 2024-09-16T11:22:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 897268736 bytes 2024-09-16T11:22:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 941138944 bytes 2024-09-16T11:22:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 991969280 bytes 2024-09-16T11:22:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1039006720 bytes 2024-09-16T11:22:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1039612928 bytes 2024-09-16T11:22:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1154671616 bytes 2024-09-16T11:22:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1201680384 bytes 2024-09-16T11:22:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1202309120 bytes 2024-09-16T11:22:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1287733248 bytes 2024-09-16T11:22:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1332149248 bytes 2024-09-16T11:22:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1354203136 bytes 2024-09-16T11:22:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1400264704 bytes 2024-09-16T11:22:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1432773632 bytes 2024-09-16T11:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1455796224 bytes 2024-09-16T11:22:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1483381760 bytes 2024-09-16T11:22:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1576744960 bytes 2024-09-16T11:22:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1577354240 bytes 2024-09-16T11:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1700193280 bytes 2024-09-16T11:22:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1748996096 bytes 2024-09-16T11:22:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1774984192 bytes 2024-09-16T11:22:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1820453888 bytes 2024-09-16T11:22:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1854610432 bytes 2024-09-16T11:22:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1949325312 bytes 2024-09-16T11:22:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2028711936 bytes 2024-09-16T11:22:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2088487936 bytes 2024-09-16T11:22:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2123418624 bytes 2024-09-16T11:22:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2185334784 bytes 2024-09-16T11:22:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2217922560 bytes 2024-09-16T11:22:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2240494592 bytes 2024-09-16T11:22:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2274549760 bytes 2024-09-16T11:22:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2366873600 bytes 2024-09-16T11:22:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2430674944 bytes 2024-09-16T11:22:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2514437120 bytes 2024-09-16T11:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2550660096 bytes 2024-09-16T11:22:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2589581312 bytes 2024-09-16T11:22:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2606675968 bytes 2024-09-16T11:22:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2664619008 bytes 2024-09-16T11:22:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2709655552 bytes 2024-09-16T11:23:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2755394560 bytes 2024-09-16T11:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2762193920 bytes 2024-09-16T11:23:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2788127744 bytes 2024-09-16T11:23:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2836492288 bytes 2024-09-16T11:23:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2842151936 bytes 2024-09-16T11:23:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2900820992 bytes 2024-09-16T11:23:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2911757312 bytes 2024-09-16T11:23:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2941270016 bytes 2024-09-16T11:23:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2976715776 bytes 2024-09-16T11:23:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2987628544 bytes 2024-09-16T11:23:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 3000228864 bytes 2024-09-16T11:23:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3000855552 bytes 2024-09-16T11:23:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3078784000 bytes 2024-09-16T11:23:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3103270912 bytes 2024-09-16T11:23:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3152775168 bytes 2024-09-16T11:23:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3183385600 bytes 2024-09-16T11:23:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3240531968 bytes 2024-09-16T11:23:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3270932480 bytes 2024-09-16T11:23:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3292055552 bytes 2024-09-16T11:23:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3369153536 bytes 2024-09-16T11:23:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3429762048 bytes 2024-09-16T11:23:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3494394880 bytes 2024-09-16T11:23:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3538305024 bytes 2024-09-16T11:23:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3543477248 bytes 2024-09-16T11:23:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3584623616 bytes 2024-09-16T11:23:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3633320960 bytes 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:23:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:23:03.841 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.994585231855996) > weight (0.25) 2024-09-16T11:23:03.842 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T11:23:03.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T11:23:03.842 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T11:23:03.842 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T11:23:03.842 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:23:03.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:23:03.840+0000 7f46629f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 41468) UID: 1000 2024-09-16T11:23:04.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:23:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":17817463.47,"bytes_per_snapshot":534837248.0,"last_snapshot_bytes":534523904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485760,"remote_snapshot_timestamp":1726485760,"replay_state":"idle"} 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:23:04 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:23:03' 2024-09-16T11:23:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:23:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:23:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:23:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":17817463.47,"bytes_per_snapshot":534837248.0,"last_snapshot_bytes":534523904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485760,"remote_snapshot_timestamp":1726485760,"replay_state":"idle"} 2024-09-16T11:23:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:23:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:23:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:23:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:23:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:23:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:23:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:23:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:23:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:23:04' 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:23:04' 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:23:04' 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:23:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='UNKNOWNUNKNOWNOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOK4438mirror.1smithi10317.2.7-1643-gd75f2382falseUNKNOWNtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":17817463.47,"bytes_per_snapshot":534837248.0,"last_snapshot_bytes":534523904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485760,"remote_snapshot_timestamp":1726485760,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:23:04cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:23:03' 2024-09-16T11:23:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:23:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:23:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:23:04' 2024-09-16T11:23:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:23:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:23:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:23:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:23:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:23:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap7 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap7 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7 2024-09-16T11:23:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:23:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:23:09.945 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T11:23:09.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:23:09.945 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:23:09.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:23:09.948 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-16T11:23:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:23:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:23:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:23:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:23:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:23:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:23:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:23:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:23:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:23:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:23:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:23:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:23:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:23:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:23:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:23:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:23:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:23:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:23:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:23:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:23:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:23:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:23:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:23:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:23:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:23:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:23:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:23:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:23:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:23:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:23:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:23:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:23:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:23:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:23:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:23:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:23:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:23:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:23:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:23:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:23:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:23:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:23:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:23:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:23:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:23:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:23:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:23:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:23:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:23:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:23:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:23:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:23:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:23:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:23:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:23:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:23:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:23:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:23:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:23:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:23:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:23:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:23:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:23:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:23:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:23:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:23:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:23:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:23:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:23:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:23:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:23:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 29 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:23:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:23:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=29 2024-09-16T11:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:23:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:23:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:23:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:23:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:23:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:23:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:23:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:23:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:23:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:23:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 29 = '' 2024-09-16T11:23:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:23:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:23:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:23:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:23:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:23:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:23:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:23:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:23:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:23:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:23:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 29 = '' 2024-09-16T11:23:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:23:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:23:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:23:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:23:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:23:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:23:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:23:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:23:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:23:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:23:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 29 = '' 2024-09-16T11:23:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:23:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:23:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:23:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:23:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=29 2024-09-16T11:23:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 29 = 29 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:23:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:23:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-16T11:23:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:23:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:23:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:23:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 14 2024-09-16T11:23:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:23:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:23:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:22:46 2024 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:23:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap6-clean ']' 2024-09-16T11:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-16T11:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6-clean 2024-09-16T11:23:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:23:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:23:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:23:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:23:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:23:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap7 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:23:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:23:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:23:44.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:23:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:23:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap7-clean 2024-09-16T11:23:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-16T11:23:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7-clean 2024-09-16T11:23:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:23:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:23:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:23:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:23:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:23:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:23:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:23:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 69918720 bytes 2024-09-16T11:23:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 81853440 bytes 2024-09-16T11:23:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 92106752 bytes 2024-09-16T11:23:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 115651584 bytes 2024-09-16T11:23:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 165407744 bytes 2024-09-16T11:23:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 283098112 bytes 2024-09-16T11:23:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 339630080 bytes 2024-09-16T11:23:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 381767680 bytes 2024-09-16T11:23:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 425359360 bytes 2024-09-16T11:23:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 464266240 bytes 2024-09-16T11:23:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 504791040 bytes 2024-09-16T11:23:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 554008576 bytes 2024-09-16T11:23:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 613466112 bytes 2024-09-16T11:23:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 711915520 bytes 2024-09-16T11:23:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 773421056 bytes 2024-09-16T11:23:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 780290048 bytes 2024-09-16T11:23:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 833175552 bytes 2024-09-16T11:23:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 854535168 bytes 2024-09-16T11:23:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 900602880 bytes 2024-09-16T11:23:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 916663296 bytes 2024-09-16T11:23:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 964218880 bytes 2024-09-16T11:23:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 969300992 bytes 2024-09-16T11:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1014411264 bytes 2024-09-16T11:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1047887872 bytes 2024-09-16T11:23:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1059602432 bytes 2024-09-16T11:23:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1078661120 bytes 2024-09-16T11:23:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1121488896 bytes 2024-09-16T11:23:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1180326912 bytes 2024-09-16T11:23:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1217435648 bytes 2024-09-16T11:23:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1239768064 bytes 2024-09-16T11:23:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1295947776 bytes 2024-09-16T11:23:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1368552448 bytes 2024-09-16T11:23:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1408658432 bytes 2024-09-16T11:23:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1486032896 bytes 2024-09-16T11:23:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1544023040 bytes 2024-09-16T11:23:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1610729472 bytes 2024-09-16T11:23:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1647175680 bytes 2024-09-16T11:23:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534837248.0,"last_snapshot_bytes":534523904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485760,"remote_snapshot_timestamp":1726485760,"replay_state":"idle"} 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:23:34 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:23:33' 2024-09-16T11:23:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:23:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:23:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:23:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534837248.0,"last_snapshot_bytes":534523904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485760,"remote_snapshot_timestamp":1726485760,"replay_state":"idle"} 2024-09-16T11:23:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:23:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:23:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:23:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:23:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:23:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:23:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:23:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:23:34' 2024-09-16T11:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:23:34' 2024-09-16T11:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:23:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:23:34' 2024-09-16T11:23:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:23:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:23:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK43224325mirror.2smithi10317.2.7-1643-gd75f2382falseOK45404543mirror.1smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534837248.0,"last_snapshot_bytes":534523904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485760,"remote_snapshot_timestamp":1726485760,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:23:34cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:23:33' 2024-09-16T11:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:23:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:23:34' 2024-09-16T11:23:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:23:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:23:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:23:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:23:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap8 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap8 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8 2024-09-16T11:23:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:24:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:24:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:24:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:24:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:24:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:24:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:24:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:24:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:24:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:24:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:24:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:24:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:24:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:24:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:24:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:24:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:24:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:24:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:24:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:24:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:24:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:24:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:24:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:24:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:24:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:24:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:24:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:24:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:24:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:24:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:24:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:24:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:24:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:24:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:24:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:24:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:24:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:24:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:24:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:24:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 33 2024-09-16T11:24:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:24:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:24:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:24:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:24:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:24:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=33 2024-09-16T11:24:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:24:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:24:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:24:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:24:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:24:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:24:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:24:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:24:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:24:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 33 = '' 2024-09-16T11:24:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:24:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:24:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:24:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:24:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:24:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:24:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:24:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:24:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:24:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 33 = '' 2024-09-16T11:24:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:24:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:24:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:24:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:24:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:24:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:24:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:24:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:24:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 33 = '' 2024-09-16T11:24:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:24:27.949 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5070957809937351) > weight (0.25) 2024-09-16T11:24:27.949 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9872725458966162) > weight (0.25) 2024-09-16T11:24:27.949 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T11:24:27.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T11:24:27.950 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T11:24:27.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:24:27.949+0000 7f4c061f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39965) UID: 1000 2024-09-16T11:24:28.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:24:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:24:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:24:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=33 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 33 = 33 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:24:28.941 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5001097034372092) > weight (0.25) 2024-09-16T11:24:28.941 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6872990082177535) > weight (0.25) 2024-09-16T11:24:28.941 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9937384675406504) > weight (0.25) 2024-09-16T11:24:28.941 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T11:24:28.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T11:24:28.942 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T11:24:28.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:24:28.941+0000 7f4b40df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 38337) UID: 1000 2024-09-16T11:24:29.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:24:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-16T11:24:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:24:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:24:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:24:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 16 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:22:46 2024 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:24:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:24:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:24:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap7-clean ']' 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7-clean 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:24:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:24:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap8 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:24:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:24:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap8-clean 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8-clean 2024-09-16T11:24:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:24:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:24:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:24:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:24:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:24:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:24:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:24:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 615424 bytes 2024-09-16T11:24:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 23132160 bytes 2024-09-16T11:24:33.942 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T11:24:33.943 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:24:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:24:33.943 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T11:24:33.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T11:24:33.944 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T11:24:33.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T11:24:33.947 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-16T11:24:33.950 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T11:24:33.951 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:24:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:24:33.951 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T11:24:33.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T11:24:33.951 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T11:24:33.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T11:24:33.954 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-16T11:24:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 66804736 bytes 2024-09-16T11:24:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 181969920 bytes 2024-09-16T11:24:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 242712576 bytes 2024-09-16T11:24:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 256629760 bytes 2024-09-16T11:24:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 280759296 bytes 2024-09-16T11:24:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 334896128 bytes 2024-09-16T11:24:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 417107968 bytes 2024-09-16T11:24:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 460913664 bytes 2024-09-16T11:24:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 554161152 bytes 2024-09-16T11:24:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 580086784 bytes 2024-09-16T11:24:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 608417792 bytes 2024-09-16T11:24:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 672787456 bytes 2024-09-16T11:24:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 742793216 bytes 2024-09-16T11:24:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 774305792 bytes 2024-09-16T11:24:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 793414656 bytes 2024-09-16T11:24:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 828882944 bytes 2024-09-16T11:24:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 860284928 bytes 2024-09-16T11:24:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 869095424 bytes 2024-09-16T11:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 902211584 bytes 2024-09-16T11:24:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 913495040 bytes 2024-09-16T11:24:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 930588672 bytes 2024-09-16T11:24:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 949535744 bytes 2024-09-16T11:24:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 974334976 bytes 2024-09-16T11:24:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 988763136 bytes 2024-09-16T11:24:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1047760896 bytes 2024-09-16T11:24:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1119327232 bytes 2024-09-16T11:24:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1146499072 bytes 2024-09-16T11:24:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1178322944 bytes 2024-09-16T11:24:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1219714048 bytes 2024-09-16T11:24:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1262978048 bytes 2024-09-16T11:24:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1289791488 bytes 2024-09-16T11:24:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1312248832 bytes 2024-09-16T11:24:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1368173568 bytes 2024-09-16T11:24:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1387936768 bytes 2024-09-16T11:24:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1433371648 bytes 2024-09-16T11:24:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1465255936 bytes 2024-09-16T11:24:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1515080704 bytes 2024-09-16T11:24:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1515727872 bytes 2024-09-16T11:24:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1524413440 bytes 2024-09-16T11:24:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1539875840 bytes 2024-09-16T11:24:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1627344896 bytes 2024-09-16T11:24:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1682518016 bytes 2024-09-16T11:24:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1701884928 bytes 2024-09-16T11:24:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1717421056 bytes 2024-09-16T11:24:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1725276160 bytes 2024-09-16T11:24:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1799433216 bytes 2024-09-16T11:24:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1829150720 bytes 2024-09-16T11:24:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1868130304 bytes 2024-09-16T11:24:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1868746752 bytes 2024-09-16T11:24:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1890405376 bytes 2024-09-16T11:24:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 1943831552 bytes 2024-09-16T11:24:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2004680704 bytes 2024-09-16T11:24:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2050412544 bytes 2024-09-16T11:24:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2069019648 bytes 2024-09-16T11:24:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2120671232 bytes 2024-09-16T11:24:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2143598592 bytes 2024-09-16T11:24:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2221895680 bytes 2024-09-16T11:24:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2243975168 bytes 2024-09-16T11:24:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2286090240 bytes 2024-09-16T11:24:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2330516480 bytes 2024-09-16T11:24:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2380826624 bytes 2024-09-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2425484288 bytes 2024-09-16T11:24:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2432467968 bytes 2024-09-16T11:24:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2437820416 bytes 2024-09-16T11:24:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2497060864 bytes 2024-09-16T11:24:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2547215360 bytes 2024-09-16T11:24:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2599996416 bytes 2024-09-16T11:24:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2643870720 bytes 2024-09-16T11:24:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2647076864 bytes 2024-09-16T11:24:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2667783168 bytes 2024-09-16T11:24:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2738928640 bytes 2024-09-16T11:24:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2812834816 bytes 2024-09-16T11:24:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 2849406976 bytes 2024-09-16T11:24:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 2895253504 bytes 2024-09-16T11:24:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 2939640832 bytes 2024-09-16T11:24:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 2973167616 bytes 2024-09-16T11:24:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3069484032 bytes 2024-09-16T11:24:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3130933248 bytes 2024-09-16T11:24:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3179484160 bytes 2024-09-16T11:24:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3217451008 bytes 2024-09-16T11:24:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3295361024 bytes 2024-09-16T11:24:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3335874560 bytes 2024-09-16T11:24:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3365685248 bytes 2024-09-16T11:24:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8600 IOs, 3415269376 bytes 2024-09-16T11:24:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8700 IOs, 3500565504 bytes 2024-09-16T11:24:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8800 IOs, 3550185472 bytes 2024-09-16T11:24:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8900 IOs, 3563365376 bytes 2024-09-16T11:24:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9000 IOs, 3644360704 bytes 2024-09-16T11:24:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9100 IOs, 3696468992 bytes 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:24:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:24:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:24:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:24:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:24:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:24:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:24:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:24:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:24:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":16966075.73,"bytes_per_snapshot":522318336.0,"last_snapshot_bytes":508982272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485864,"remote_snapshot_timestamp":1726485864,"replay_state":"idle"} 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:24:34 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:24:33' 2024-09-16T11:24:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:24:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:24:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":16966075.73,"bytes_per_snapshot":522318336.0,"last_snapshot_bytes":508982272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485864,"remote_snapshot_timestamp":1726485864,"replay_state":"idle"} 2024-09-16T11:24:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:24:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:24:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:24:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:24:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:24:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:24:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:24:34' 2024-09-16T11:24:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:34' 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:24:34' 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:24:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK45404543mirror.1smithi10317.2.7-1643-gd75f2382falseOK45984604mirror.2smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":16966075.73,"bytes_per_snapshot":522318336.0,"last_snapshot_bytes":508982272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485864,"remote_snapshot_timestamp":1726485864,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:24:34cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:24:33' 2024-09-16T11:24:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:24:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:24:34' 2024-09-16T11:24:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:24:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:24:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:24:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:24:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:24:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap9 2024-09-16T11:24:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap9 2024-09-16T11:24:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:24:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:24:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:24:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9 2024-09-16T11:24:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:25:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:25:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:25:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:25:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:25:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:25:18.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:25:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:25:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:25:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:25:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:25:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:25:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:25:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:25:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:25:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:25:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:25:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:25:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:25:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:25:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:25:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:25:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:25:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:25:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:25:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:25:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:25:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:25:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:25:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:25:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:25:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:25:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:25:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:25:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 37 2024-09-16T11:25:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:25:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:25:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:25:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:25:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:25:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=37 2024-09-16T11:25:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:25:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:25:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:25:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:25:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:25:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:25:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:25:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:25:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:25:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 37 = '' 2024-09-16T11:25:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:25:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:25:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:25:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:25:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 37 = '' 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:25:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:25:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:25:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:25:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:25:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:25:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:25:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:25:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:25:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 37 = '' 2024-09-16T11:25:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:25:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:25:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:25:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:25:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:25:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=37 2024-09-16T11:25:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 37 = 37 2024-09-16T11:25:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:25:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:25:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 18 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:24:30 2024 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap8-clean ']' 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8-clean 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:25:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:25:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:25:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap9 2024-09-16T11:25:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9 2024-09-16T11:25:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:25:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:25:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:25:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:25:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:25:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:25:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap9-clean 2024-09-16T11:25:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-16T11:25:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9-clean 2024-09-16T11:25:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T11:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:25:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:25:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:25:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T11:25:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 39443456 bytes 2024-09-16T11:25:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 40027136 bytes 2024-09-16T11:25:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 125819904 bytes 2024-09-16T11:25:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 168958976 bytes 2024-09-16T11:25:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 230860800 bytes 2024-09-16T11:25:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 271567872 bytes 2024-09-16T11:25:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 300518400 bytes 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 329454592 bytes 2024-09-16T11:25:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 399239168 bytes 2024-09-16T11:25:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 444464128 bytes 2024-09-16T11:25:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 482618368 bytes 2024-09-16T11:25:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 506320896 bytes 2024-09-16T11:25:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 600357888 bytes 2024-09-16T11:25:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 617355264 bytes 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 669184000 bytes 2024-09-16T11:25:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 752733184 bytes 2024-09-16T11:25:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 787222528 bytes 2024-09-16T11:25:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 826300416 bytes 2024-09-16T11:25:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 908120064 bytes 2024-09-16T11:25:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 965584896 bytes 2024-09-16T11:25:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 992551936 bytes 2024-09-16T11:25:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1045955584 bytes 2024-09-16T11:25:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1077103616 bytes 2024-09-16T11:25:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1117186048 bytes 2024-09-16T11:25:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1144822784 bytes 2024-09-16T11:25:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1195899904 bytes 2024-09-16T11:25:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1210311680 bytes 2024-09-16T11:25:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1221562368 bytes 2024-09-16T11:25:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1244342272 bytes 2024-09-16T11:25:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1280449536 bytes 2024-09-16T11:25:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1300519936 bytes 2024-09-16T11:25:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1301131264 bytes 2024-09-16T11:25:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:25:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:25:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:25:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:25:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local description_pattern= 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local service_pattern= 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:25:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='test: 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: global_id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+replaying 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: replaying, {"bytes_per_second":17874739.2,"bytes_per_snapshot":522612224.0,"last_snapshot_bytes":536242176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485919,"remote_snapshot_timestamp":1726485919,"replay_state":"idle"} 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: service: mirror.3 on smithi103 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:25:34 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: peer_sites: 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: name: cluster2 2024-09-16T11:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: state: up+stopped 2024-09-16T11:25:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: description: local image is primary 2024-09-16T11:25:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: last_update: 2024-09-16 11:25:33' 2024-09-16T11:25:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: state: up+replaying 2024-09-16T11:25:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^ description: .*' 2024-09-16T11:25:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: description: replaying, {"bytes_per_second":17874739.2,"bytes_per_snapshot":522612224.0,"last_snapshot_bytes":536242176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485919,"remote_snapshot_timestamp":1726485919,"replay_state":"idle"} 2024-09-16T11:25:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:25:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:25:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep '^up+' 2024-09-16T11:25:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:25:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: service: mirror.3 on smithi103 2024-09-16T11:25:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update 2024-09-16T11:25:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:25:34' 2024-09-16T11:25:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:25:34' 2024-09-16T11:25:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:25:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local 'prev_last_update=2024-09-16 11:25:34' 2024-09-16T11:25:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status 2024-09-16T11:25:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ CEPH_ARGS= 2024-09-16T11:25:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status='OKOKOK142104293mirror.0smithi10317.2.7-1643-gd75f2382trueOK42224288mirror.3smithi10317.2.7-1643-gd75f2382falseOK45404543mirror.1smithi10317.2.7-1643-gd75f2382falseOK45984604mirror.2smithi10317.2.7-1643-gd75f2382falseOKtestc00baae6-29b2-4df0-b664-37b080b9c366up+replayingreplaying, {"bytes_per_second":17874739.2,"bytes_per_snapshot":522612224.0,"last_snapshot_bytes":536242176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726485919,"remote_snapshot_timestamp":1726485919,"replay_state":"idle"}42224288mirror.3smithi1032024-09-16 11:25:34cluster204869fab-2245-439f-841b-481bc7c8cad8up+stoppedlocal image is primary2024-09-16 11:25:33' 2024-09-16T11:25:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_update state 2024-09-16T11:25:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ last_update='2024-09-16 11:25:34' 2024-09-16T11:25:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=up+replaying 2024-09-16T11:25:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo up+replaying 2024-09-16T11:25:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep up+replaying 2024-09-16T11:25:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:up+replaying 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap10 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap10 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10 2024-09-16T11:25:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:25:41.948 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.47128353728078) > weight (0.25) 2024-09-16T11:25:41.949 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9662053898132957) > weight (0.25) 2024-09-16T11:25:41.949 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T11:25:41.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T11:25:41.949 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T11:25:41.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:25:41.948+0000 7f19de3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42186) UID: 1000 2024-09-16T11:25:42.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:25:42.950 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T11:25:42.950 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:25:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:25:42.950 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T11:25:42.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T11:25:42.951 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T11:25:42.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T11:25:42.954 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-16T11:26:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:26:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:26:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:26:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:26:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:26:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:26:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:26:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:26:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:26:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:26:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:26:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:26:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:26:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:26:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:26:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:26:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:26:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:26:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:26:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:26:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:26:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:26:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:26:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:26:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:26:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:26:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:26:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:26:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:26:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:26:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:26:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:26:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:26:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:26:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:26:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:26:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:26:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:26:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:26:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 41 2024-09-16T11:26:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:26:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:26:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:26:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=41 2024-09-16T11:26:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:26:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:26:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:26:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:26:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 41 = '' 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:26:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:26:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:26:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:26:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:26:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:26:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:26:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 41 = '' 2024-09-16T11:26:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:26:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:26:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:26:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:26:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:26:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:26:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:26:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:26:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 41 = '' 2024-09-16T11:26:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:26:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:26:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=41 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 41 = 41 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:26:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:26:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 20 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:26:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:24:30 2024 2024-09-16T11:26:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:26:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:26:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:26:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:26:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:26:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snap9-clean ']' 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9-clean 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap10 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ clean_snap_name=snap10-clean 2024-09-16T11:26:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-16T11:26:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10-clean 2024-09-16T11:26:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T11:26:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:26:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:26:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:26:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:26:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:26:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:26:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:26:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:26:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:26:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:26:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:26:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:26:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:26:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T11:26:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:26:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:26:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:26:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:26:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:26:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:26:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:26:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T11:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:26:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:26:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:26:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:26:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:26:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:26:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:26:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:26:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T11:26:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:26:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:26:17.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-test.status 2024-09-16T11:26:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:26:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T11:26:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-test.status 2024-09-16T11:26:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T11:26:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T11:26:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 44 2024-09-16T11:26:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:26:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.status 2024-09-16T11:26:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T11:26:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:26:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=44 2024-09-16T11:26:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:26:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:26:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:26:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:26:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:26:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:26:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:26:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:26:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:26:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 44 = '' 2024-09-16T11:26:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:26:19.954 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T11:26:19.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T11:26:19.955 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T11:26:19.955 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T11:26:19.955 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:26:19.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:26:19.954+0000 7f379f7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 38125) UID: 1000 2024-09-16T11:26:20.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.status 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T11:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:26:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=44 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 44 = 44 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10-clean 2024-09-16T11:26:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:26:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:26:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:26:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'test': 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 21 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10bf4eaeea0 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10bf4eaeea0 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:26:13 2024 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:17:36 2024 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c00baae6-29b2-4df0-b664-37b080b9c366 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 10 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap1 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap1 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap1-clean 2024-09-16T11:26:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-16T11:26:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap1-clean 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap2 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap2 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:26:26.057 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T11:26:26.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:26:26.057 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:26:26.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:26:26.060 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T11:26:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap2-clean 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap2-clean 2024-09-16T11:26:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap3 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap3 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap3-clean 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap3-clean 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap4 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap4 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4 2024-09-16T11:26:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap4-clean 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap4-clean 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap5 2024-09-16T11:26:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap5 2024-09-16T11:26:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5 2024-09-16T11:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap5-clean 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap5-clean 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap6 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap6 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6 2024-09-16T11:26:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap6-clean 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap6-clean 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap7 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap7 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7 2024-09-16T11:26:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap7-clean 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap7-clean 2024-09-16T11:26:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap8 2024-09-16T11:26:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap8 2024-09-16T11:26:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap8-clean 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap8-clean 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap9 2024-09-16T11:26:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap9 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap9-clean 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap9-clean 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap10 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap10 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10 2024-09-16T11:26:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap10-clean 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap10-clean 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export 2024-09-16T11:26:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-test.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-test.export 2024-09-16T11:26:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:26:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 10 2024-09-16T11:26:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap1 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap1 2024-09-16T11:26:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-16T11:26:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:26:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap1-clean 2024-09-16T11:26:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-16T11:26:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap1-clean 2024-09-16T11:26:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-16T11:26:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:26:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap2 2024-09-16T11:26:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-16T11:26:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap2 2024-09-16T11:26:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-16T11:26:56.955 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9761584337626759) > weight (0.25) 2024-09-16T11:26:56.955 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T11:26:56.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T11:26:56.955 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T11:26:56.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:26:56.955+0000 7f52883f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41014) UID: 1000 2024-09-16T11:26:57.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:26:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap2-clean 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap2-clean 2024-09-16T11:26:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-16T11:26:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap3 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap3 2024-09-16T11:26:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-16T11:26:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap3-clean 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap3-clean 2024-09-16T11:26:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-16T11:26:59.956 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T11:26:59.956 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:26:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:26:59.957 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T11:26:59.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T11:26:59.957 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T11:26:59.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T11:26:59.960 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-16T11:27:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap4 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap4 2024-09-16T11:27:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-16T11:27:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap4-clean 2024-09-16T11:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-16T11:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap4-clean 2024-09-16T11:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-16T11:27:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap5 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap5 2024-09-16T11:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-16T11:27:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap5-clean 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap5-clean 2024-09-16T11:27:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-16T11:27:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap6 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap6 2024-09-16T11:27:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-16T11:27:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap6-clean 2024-09-16T11:27:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-16T11:27:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap6-clean 2024-09-16T11:27:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-16T11:27:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap7 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap7 2024-09-16T11:27:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-16T11:27:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap7-clean 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap7-clean 2024-09-16T11:27:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-16T11:27:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap8 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap8 2024-09-16T11:27:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-16T11:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap8-clean 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap8-clean 2024-09-16T11:27:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-16T11:27:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap9 2024-09-16T11:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-16T11:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap9 2024-09-16T11:27:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-16T11:27:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap9-clean 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap9-clean 2024-09-16T11:27:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-16T11:27:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap10 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap10 2024-09-16T11:27:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-16T11:27:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap10-clean 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap10-clean 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-16T11:27:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror test 2024-09-16T11:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror test 2024-09-16T11:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-16T11:27:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-16T11:27:14.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:27:14.834+0000 7f348ffea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559a27273ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:27:14.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:27:14.834+0000 7f3496ff8640 -1 rbd::mirror::ImageReplayer: 0x559a23d74280 [3/c00baae6-29b2-4df0-b664-37b080b9c366] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:27:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:27:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:27:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=test 2024-09-16T11:27:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:27:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:27:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:27:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:27:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:27:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=test 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:27:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=test 2024-09-16T11:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:27:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:27:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: create many images' 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:27:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:27:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:27:16 TEST: create many images 2024-09-16T11:27:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:27:16 TEST: create many images 2024-09-16T11:27:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ snap_name=snap 2024-09-16T11:27:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 50 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_1 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:27:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:27:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-16T11:27:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:27:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-16T11:27:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_1': 2024-09-16T11:27:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12801bd3cc99 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12801bd3cc99 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:16 2024 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:16 2024 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:16 2024 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:27:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:27:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: dca53849-69bc-40ac-a731-5bbd18f66953 2024-09-16T11:27:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:27:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-16T11:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:27:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:27:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=10 2024-09-16T11:27:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:27:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:27:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 83389440 bytes 2024-09-16T11:27:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 123212800 bytes 2024-09-16T11:27:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 138518528 bytes 2024-09-16T11:27:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 258657280 bytes 2024-09-16T11:27:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 288625664 bytes 2024-09-16T11:27:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 333322240 bytes 2024-09-16T11:27:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 358066176 bytes 2024-09-16T11:27:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 400641024 bytes 2024-09-16T11:27:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 487499776 bytes 2024-09-16T11:27:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 530026496 bytes 2024-09-16T11:27:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 585585664 bytes 2024-09-16T11:27:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 667471872 bytes 2024-09-16T11:27:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 737133568 bytes 2024-09-16T11:27:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 802440192 bytes 2024-09-16T11:27:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 862572544 bytes 2024-09-16T11:27:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 909149184 bytes 2024-09-16T11:27:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 933494784 bytes 2024-09-16T11:27:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 988215296 bytes 2024-09-16T11:27:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1036395520 bytes 2024-09-16T11:27:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1058634752 bytes 2024-09-16T11:27:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1109113856 bytes 2024-09-16T11:27:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1126216704 bytes 2024-09-16T11:27:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1161318400 bytes 2024-09-16T11:27:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1165084672 bytes 2024-09-16T11:27:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1227361280 bytes 2024-09-16T11:27:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1236786176 bytes 2024-09-16T11:27:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1271001088 bytes 2024-09-16T11:27:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1335820288 bytes 2024-09-16T11:27:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1343666176 bytes 2024-09-16T11:27:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1406147584 bytes 2024-09-16T11:27:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1455660032 bytes 2024-09-16T11:27:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1470801920 bytes 2024-09-16T11:27:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1497334784 bytes 2024-09-16T11:27:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1531216896 bytes 2024-09-16T11:27:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1599183872 bytes 2024-09-16T11:27:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1633356800 bytes 2024-09-16T11:27:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1723974656 bytes 2024-09-16T11:27:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1779980288 bytes 2024-09-16T11:27:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1806621696 bytes 2024-09-16T11:27:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1836931072 bytes 2024-09-16T11:27:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1877690368 bytes 2024-09-16T11:27:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1897212928 bytes 2024-09-16T11:27:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1913908224 bytes 2024-09-16T11:27:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1958598656 bytes 2024-09-16T11:27:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2010235904 bytes 2024-09-16T11:27:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2119860224 bytes 2024-09-16T11:27:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2160151552 bytes 2024-09-16T11:27:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2226257920 bytes 2024-09-16T11:27:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2257018880 bytes 2024-09-16T11:27:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2333176832 bytes 2024-09-16T11:27:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2411599872 bytes 2024-09-16T11:27:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2470729728 bytes 2024-09-16T11:27:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2500161536 bytes 2024-09-16T11:27:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2534760448 bytes 2024-09-16T11:27:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2596389888 bytes 2024-09-16T11:27:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2621785088 bytes 2024-09-16T11:27:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2625345536 bytes 2024-09-16T11:27:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2642250752 bytes 2024-09-16T11:27:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2686961664 bytes 2024-09-16T11:27:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2687564800 bytes 2024-09-16T11:27:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2721085440 bytes 2024-09-16T11:27:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2748913664 bytes 2024-09-16T11:27:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2792157184 bytes 2024-09-16T11:27:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:27:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:27:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:27:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_2 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-16T11:27:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:27:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_2': 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 128c641d81d7 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.128c641d81d7 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:26 2024 2024-09-16T11:27:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:26 2024 2024-09-16T11:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:26 2024 2024-09-16T11:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c33e6270-8531-45ef-bdc4-ecc6d15eb898 2024-09-16T11:27:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:27:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:27:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-16T11:27:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:27:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:27:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:27:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:27:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:27:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 139853824 bytes 2024-09-16T11:27:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 150733824 bytes 2024-09-16T11:27:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 224203776 bytes 2024-09-16T11:27:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 263624704 bytes 2024-09-16T11:27:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 346990592 bytes 2024-09-16T11:27:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 400669696 bytes 2024-09-16T11:27:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 460677120 bytes 2024-09-16T11:27:29.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 472341504 bytes 2024-09-16T11:27:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 536858624 bytes 2024-09-16T11:27:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 552933376 bytes 2024-09-16T11:27:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 572107776 bytes 2024-09-16T11:27:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 596675584 bytes 2024-09-16T11:27:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 647238656 bytes 2024-09-16T11:27:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 661768192 bytes 2024-09-16T11:27:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 681771008 bytes 2024-09-16T11:27:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 790215680 bytes 2024-09-16T11:27:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 843039744 bytes 2024-09-16T11:27:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 887115776 bytes 2024-09-16T11:27:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 896232448 bytes 2024-09-16T11:27:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 980833280 bytes 2024-09-16T11:27:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 981427200 bytes 2024-09-16T11:27:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 998966272 bytes 2024-09-16T11:27:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1021798400 bytes 2024-09-16T11:27:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1022408704 bytes 2024-09-16T11:27:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1058900992 bytes 2024-09-16T11:27:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1109760000 bytes 2024-09-16T11:27:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1120486400 bytes 2024-09-16T11:27:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1178288128 bytes 2024-09-16T11:27:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1253575680 bytes 2024-09-16T11:27:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1266180096 bytes 2024-09-16T11:27:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1278835712 bytes 2024-09-16T11:27:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1279460352 bytes 2024-09-16T11:27:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1314453504 bytes 2024-09-16T11:27:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1372880896 bytes 2024-09-16T11:27:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1436896256 bytes 2024-09-16T11:27:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1487222784 bytes 2024-09-16T11:27:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1568865280 bytes 2024-09-16T11:27:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1569503232 bytes 2024-09-16T11:27:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1616610304 bytes 2024-09-16T11:27:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1620189184 bytes 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_3 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-16T11:27:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:27:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:27:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:27:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:27:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2024-09-16T11:27:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:27:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-16T11:27:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:27:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-16T11:27:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:27:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-16T11:27:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_3': 2024-09-16T11:27:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1298f4657121 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1298f4657121 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:33 2024 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:33 2024 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:33 2024 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:27:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 4b1d8b43-03c2-4e7d-bdc2-68e83df2c512 2024-09-16T11:27:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:27:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:27:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-16T11:27:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:27:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:27:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:27:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:27:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:27:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 56974336 bytes 2024-09-16T11:27:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 106680320 bytes 2024-09-16T11:27:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 147315712 bytes 2024-09-16T11:27:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 213989376 bytes 2024-09-16T11:27:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 277288960 bytes 2024-09-16T11:27:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 317236224 bytes 2024-09-16T11:27:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 386379776 bytes 2024-09-16T11:27:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 414277632 bytes 2024-09-16T11:27:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 456756224 bytes 2024-09-16T11:27:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 492875776 bytes 2024-09-16T11:27:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 599271424 bytes 2024-09-16T11:27:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 653822976 bytes 2024-09-16T11:27:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 699323392 bytes 2024-09-16T11:27:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 768442368 bytes 2024-09-16T11:27:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 828321792 bytes 2024-09-16T11:27:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 914195456 bytes 2024-09-16T11:27:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 1009734656 bytes 2024-09-16T11:27:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 1040353280 bytes 2024-09-16T11:27:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1095425024 bytes 2024-09-16T11:27:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1155261440 bytes 2024-09-16T11:27:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1210616832 bytes 2024-09-16T11:27:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1291049984 bytes 2024-09-16T11:27:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1362573312 bytes 2024-09-16T11:27:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1404206080 bytes 2024-09-16T11:27:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1455537152 bytes 2024-09-16T11:27:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1555110912 bytes 2024-09-16T11:27:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1593060352 bytes 2024-09-16T11:27:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1673709568 bytes 2024-09-16T11:27:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1674310656 bytes 2024-09-16T11:27:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1710342144 bytes 2024-09-16T11:27:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1741050880 bytes 2024-09-16T11:27:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1786309632 bytes 2024-09-16T11:27:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1794605056 bytes 2024-09-16T11:27:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1808344064 bytes 2024-09-16T11:27:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1829055488 bytes 2024-09-16T11:27:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1869189120 bytes 2024-09-16T11:27:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1888293888 bytes 2024-09-16T11:27:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:27:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_4 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-16T11:27:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:27:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_4': 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a4faf56688 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a4faf56688 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:40 2024 2024-09-16T11:27:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:40 2024 2024-09-16T11:27:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:40 2024 2024-09-16T11:27:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:27:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:27:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 18f01d1c-b747-4fab-8897-4595d20eb919 2024-09-16T11:27:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:27:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:27:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-16T11:27:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:27:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:27:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=13 2024-09-16T11:27:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:27:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:27:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 69997568 bytes 2024-09-16T11:27:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 81373184 bytes 2024-09-16T11:27:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 83331072 bytes 2024-09-16T11:27:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 194711552 bytes 2024-09-16T11:27:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 256906240 bytes 2024-09-16T11:27:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 315038720 bytes 2024-09-16T11:27:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 395691008 bytes 2024-09-16T11:27:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 445645824 bytes 2024-09-16T11:27:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 520627200 bytes 2024-09-16T11:27:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 559471616 bytes 2024-09-16T11:27:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 586384384 bytes 2024-09-16T11:27:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 643454976 bytes 2024-09-16T11:27:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 669238272 bytes 2024-09-16T11:27:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 759282688 bytes 2024-09-16T11:27:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 791260160 bytes 2024-09-16T11:27:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 861335552 bytes 2024-09-16T11:27:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 937043968 bytes 2024-09-16T11:27:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 970070016 bytes 2024-09-16T11:27:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 992480256 bytes 2024-09-16T11:27:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1014155264 bytes 2024-09-16T11:27:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1035073536 bytes 2024-09-16T11:27:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1084158976 bytes 2024-09-16T11:27:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1161269248 bytes 2024-09-16T11:27:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1199495168 bytes 2024-09-16T11:27:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1232325632 bytes 2024-09-16T11:27:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1262887936 bytes 2024-09-16T11:27:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1282512896 bytes 2024-09-16T11:27:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1378640896 bytes 2024-09-16T11:27:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1440740352 bytes 2024-09-16T11:27:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1459944448 bytes 2024-09-16T11:27:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1481252864 bytes 2024-09-16T11:27:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1507602432 bytes 2024-09-16T11:27:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1529565184 bytes 2024-09-16T11:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1582580736 bytes 2024-09-16T11:27:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1614877696 bytes 2024-09-16T11:27:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1640627200 bytes 2024-09-16T11:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1693287424 bytes 2024-09-16T11:27:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1729602560 bytes 2024-09-16T11:27:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1741310976 bytes 2024-09-16T11:27:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1775749120 bytes 2024-09-16T11:27:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1786869760 bytes 2024-09-16T11:27:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1825254400 bytes 2024-09-16T11:27:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1906523136 bytes 2024-09-16T11:27:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1936558080 bytes 2024-09-16T11:27:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1969474560 bytes 2024-09-16T11:27:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2032749568 bytes 2024-09-16T11:27:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2112699392 bytes 2024-09-16T11:27:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2133531648 bytes 2024-09-16T11:27:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2161905664 bytes 2024-09-16T11:27:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2235981824 bytes 2024-09-16T11:27:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2290294784 bytes 2024-09-16T11:27:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2346584064 bytes 2024-09-16T11:27:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2421464064 bytes 2024-09-16T11:27:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2464931840 bytes 2024-09-16T11:27:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2502912000 bytes 2024-09-16T11:27:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2563169280 bytes 2024-09-16T11:27:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2602118144 bytes 2024-09-16T11:27:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2643491840 bytes 2024-09-16T11:27:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2698072064 bytes 2024-09-16T11:27:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2698679296 bytes 2024-09-16T11:27:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2737438720 bytes 2024-09-16T11:27:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2748261376 bytes 2024-09-16T11:27:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2773638144 bytes 2024-09-16T11:27:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2834660352 bytes 2024-09-16T11:27:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2861277184 bytes 2024-09-16T11:27:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2861882368 bytes 2024-09-16T11:27:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2877682688 bytes 2024-09-16T11:27:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2894901248 bytes 2024-09-16T11:27:52.061 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6253061602663197) > weight (0.25) 2024-09-16T11:27:52.061 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4001989183490807) > weight (0.25) 2024-09-16T11:27:52.061 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5437488009410661) > weight (0.25) 2024-09-16T11:27:52.061 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.41219025688025235) > weight (0.25) 2024-09-16T11:27:52.061 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-16T11:27:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2970583040 bytes 2024-09-16T11:27:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3021617152 bytes 2024-09-16T11:27:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3086313472 bytes 2024-09-16T11:27:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3110289408 bytes 2024-09-16T11:27:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3157469184 bytes 2024-09-16T11:27:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3189322752 bytes 2024-09-16T11:27:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3256664064 bytes 2024-09-16T11:27:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3315297280 bytes 2024-09-16T11:27:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3346836480 bytes 2024-09-16T11:27:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3402434560 bytes 2024-09-16T11:27:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3403067392 bytes 2024-09-16T11:27:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3515879424 bytes 2024-09-16T11:27:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3516504064 bytes 2024-09-16T11:27:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3534926848 bytes 2024-09-16T11:27:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3595100160 bytes 2024-09-16T11:27:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3601232896 bytes 2024-09-16T11:27:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3661111296 bytes 2024-09-16T11:27:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:27:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:27:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:27:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:27:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:27:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_5 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:27:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2024-09-16T11:27:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:27:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-16T11:27:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:27:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:27:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-16T11:27:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:27:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_5': 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12b0c697c2e9 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12b0c697c2e9 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:27:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:54 2024 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:54 2024 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:54 2024 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 735bbf09-b5b2-4a17-8414-9de64f208e63 2024-09-16T11:27:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:27:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:27:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-16T11:27:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:27:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:27:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:27:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:27:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:27:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:27:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:27:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 29370368 bytes 2024-09-16T11:27:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 37995520 bytes 2024-09-16T11:27:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 83852288 bytes 2024-09-16T11:27:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 105345024 bytes 2024-09-16T11:27:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 132240384 bytes 2024-09-16T11:27:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 163451904 bytes 2024-09-16T11:27:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 216899584 bytes 2024-09-16T11:27:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 230934528 bytes 2024-09-16T11:27:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 273597440 bytes 2024-09-16T11:27:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 312879104 bytes 2024-09-16T11:27:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 378621952 bytes 2024-09-16T11:27:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 387971072 bytes 2024-09-16T11:27:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 468013056 bytes 2024-09-16T11:27:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 511462400 bytes 2024-09-16T11:27:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 542191616 bytes 2024-09-16T11:27:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 565241856 bytes 2024-09-16T11:27:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 591946752 bytes 2024-09-16T11:27:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 628186112 bytes 2024-09-16T11:27:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 673492992 bytes 2024-09-16T11:27:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 675309568 bytes 2024-09-16T11:27:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 713303040 bytes 2024-09-16T11:27:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 742053888 bytes 2024-09-16T11:27:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 756651008 bytes 2024-09-16T11:27:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 784160768 bytes 2024-09-16T11:27:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 808281088 bytes 2024-09-16T11:27:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 821050368 bytes 2024-09-16T11:27:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 891079680 bytes 2024-09-16T11:27:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 916202496 bytes 2024-09-16T11:27:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 963132416 bytes 2024-09-16T11:27:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 981178368 bytes 2024-09-16T11:27:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1027352576 bytes 2024-09-16T11:27:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1063468032 bytes 2024-09-16T11:27:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1118520320 bytes 2024-09-16T11:27:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1119138816 bytes 2024-09-16T11:27:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1143457792 bytes 2024-09-16T11:27:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1219798016 bytes 2024-09-16T11:27:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1256914944 bytes 2024-09-16T11:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1310970880 bytes 2024-09-16T11:28:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:28:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:28:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:28:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:28:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:28:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_6 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2024-09-16T11:28:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:28:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-16T11:28:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:28:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-16T11:28:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_6': 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12bcb3d0af79 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12bcb3d0af79 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:28:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:00 2024 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:00 2024 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:00 2024 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: bd2d7a6e-4890-4807-895c-c7d327ef0b8d 2024-09-16T11:28:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-16T11:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:28:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=10 2024-09-16T11:28:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:28:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:28:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 696320 bytes 2024-09-16T11:28:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 30989312 bytes 2024-09-16T11:28:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 74668032 bytes 2024-09-16T11:28:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 107387904 bytes 2024-09-16T11:28:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 134035456 bytes 2024-09-16T11:28:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 178033664 bytes 2024-09-16T11:28:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 257685504 bytes 2024-09-16T11:28:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 333023232 bytes 2024-09-16T11:28:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 367594496 bytes 2024-09-16T11:28:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 380577792 bytes 2024-09-16T11:28:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 436023296 bytes 2024-09-16T11:28:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 475609088 bytes 2024-09-16T11:28:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 552318976 bytes 2024-09-16T11:28:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 618874880 bytes 2024-09-16T11:28:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 675965952 bytes 2024-09-16T11:28:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 752208896 bytes 2024-09-16T11:28:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 819022848 bytes 2024-09-16T11:28:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 869734400 bytes 2024-09-16T11:28:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 920673280 bytes 2024-09-16T11:28:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 964874240 bytes 2024-09-16T11:28:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1029663744 bytes 2024-09-16T11:28:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1062885376 bytes 2024-09-16T11:28:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1114034176 bytes 2024-09-16T11:28:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1132960768 bytes 2024-09-16T11:28:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1133596672 bytes 2024-09-16T11:28:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1164218368 bytes 2024-09-16T11:28:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1202184192 bytes 2024-09-16T11:28:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1215453184 bytes 2024-09-16T11:28:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1236627456 bytes 2024-09-16T11:28:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1288279040 bytes 2024-09-16T11:28:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1307597824 bytes 2024-09-16T11:28:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1340569600 bytes 2024-09-16T11:28:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1424206848 bytes 2024-09-16T11:28:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1450575872 bytes 2024-09-16T11:28:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1482131456 bytes 2024-09-16T11:28:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1544772608 bytes 2024-09-16T11:28:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1571856384 bytes 2024-09-16T11:28:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1587350528 bytes 2024-09-16T11:28:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1649342464 bytes 2024-09-16T11:28:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1727059968 bytes 2024-09-16T11:28:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1736124416 bytes 2024-09-16T11:28:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1818719232 bytes 2024-09-16T11:28:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1894092800 bytes 2024-09-16T11:28:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1912427520 bytes 2024-09-16T11:28:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1962668032 bytes 2024-09-16T11:28:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2055233536 bytes 2024-09-16T11:28:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2109844480 bytes 2024-09-16T11:28:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2137306112 bytes 2024-09-16T11:28:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2144954368 bytes 2024-09-16T11:28:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2190813184 bytes 2024-09-16T11:28:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2266827776 bytes 2024-09-16T11:28:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2295447552 bytes 2024-09-16T11:28:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2340487168 bytes 2024-09-16T11:28:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2383371264 bytes 2024-09-16T11:28:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2415625216 bytes 2024-09-16T11:28:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2420281344 bytes 2024-09-16T11:28:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2446774272 bytes 2024-09-16T11:28:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2489089024 bytes 2024-09-16T11:28:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2513611776 bytes 2024-09-16T11:28:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2551096320 bytes 2024-09-16T11:28:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2675012608 bytes 2024-09-16T11:28:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2720886784 bytes 2024-09-16T11:28:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2784502784 bytes 2024-09-16T11:28:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2865451008 bytes 2024-09-16T11:28:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2916020224 bytes 2024-09-16T11:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2967697408 bytes 2024-09-16T11:28:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 3001452544 bytes 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_7 2024-09-16T11:28:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:28:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:28:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:28:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:28:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-16T11:28:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:28:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_7': 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12c8cef5c602 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12c8cef5c602 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:11 2024 2024-09-16T11:28:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:11 2024 2024-09-16T11:28:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:11 2024 2024-09-16T11:28:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:28:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:28:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 215e6e09-36a7-4867-8fa8-8c5aab26f0d4 2024-09-16T11:28:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:28:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:28:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-16T11:28:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:28:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:28:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=11 2024-09-16T11:28:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:28:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:28:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 18513920 bytes 2024-09-16T11:28:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 62492672 bytes 2024-09-16T11:28:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 119530496 bytes 2024-09-16T11:28:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 140696576 bytes 2024-09-16T11:28:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 233561088 bytes 2024-09-16T11:28:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 303724544 bytes 2024-09-16T11:28:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 364599296 bytes 2024-09-16T11:28:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 389138432 bytes 2024-09-16T11:28:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 468733952 bytes 2024-09-16T11:28:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 491528192 bytes 2024-09-16T11:28:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 540278784 bytes 2024-09-16T11:28:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 540895232 bytes 2024-09-16T11:28:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 558034944 bytes 2024-09-16T11:28:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 587439104 bytes 2024-09-16T11:28:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 733251584 bytes 2024-09-16T11:28:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 768457728 bytes 2024-09-16T11:28:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 788119552 bytes 2024-09-16T11:28:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 788738048 bytes 2024-09-16T11:28:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 858356736 bytes 2024-09-16T11:28:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 877163520 bytes 2024-09-16T11:28:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 923090944 bytes 2024-09-16T11:28:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 973756416 bytes 2024-09-16T11:28:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 997210112 bytes 2024-09-16T11:28:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1017787392 bytes 2024-09-16T11:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1071017984 bytes 2024-09-16T11:28:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1180564480 bytes 2024-09-16T11:28:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1230073856 bytes 2024-09-16T11:28:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1274977280 bytes 2024-09-16T11:28:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1332268032 bytes 2024-09-16T11:28:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1366362112 bytes 2024-09-16T11:28:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1431858176 bytes 2024-09-16T11:28:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1482156032 bytes 2024-09-16T11:28:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1539444736 bytes 2024-09-16T11:28:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1619961856 bytes 2024-09-16T11:28:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1678885888 bytes 2024-09-16T11:28:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1737585664 bytes 2024-09-16T11:28:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1744202752 bytes 2024-09-16T11:28:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1753266176 bytes 2024-09-16T11:28:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1785108480 bytes 2024-09-16T11:28:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1825465344 bytes 2024-09-16T11:28:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1895764992 bytes 2024-09-16T11:28:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1899169792 bytes 2024-09-16T11:28:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1919503360 bytes 2024-09-16T11:28:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1938504704 bytes 2024-09-16T11:28:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1974224896 bytes 2024-09-16T11:28:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2058984448 bytes 2024-09-16T11:28:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2085266432 bytes 2024-09-16T11:28:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2140193792 bytes 2024-09-16T11:28:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2176157696 bytes 2024-09-16T11:28:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2176781312 bytes 2024-09-16T11:28:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2225459200 bytes 2024-09-16T11:28:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2231606272 bytes 2024-09-16T11:28:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2232238080 bytes 2024-09-16T11:28:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2285714432 bytes 2024-09-16T11:28:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2311403520 bytes 2024-09-16T11:28:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2329027584 bytes 2024-09-16T11:28:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2343938048 bytes 2024-09-16T11:28:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2370343936 bytes 2024-09-16T11:28:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2398289920 bytes 2024-09-16T11:28:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2418217984 bytes 2024-09-16T11:28:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2470082560 bytes 2024-09-16T11:28:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2478035968 bytes 2024-09-16T11:28:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2542088192 bytes 2024-09-16T11:28:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2552189952 bytes 2024-09-16T11:28:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2584981504 bytes 2024-09-16T11:28:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2640130048 bytes 2024-09-16T11:28:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2674717696 bytes 2024-09-16T11:28:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2762854400 bytes 2024-09-16T11:28:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2794143744 bytes 2024-09-16T11:28:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2842917888 bytes 2024-09-16T11:28:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2880292864 bytes 2024-09-16T11:28:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2976052224 bytes 2024-09-16T11:28:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2984150016 bytes 2024-09-16T11:28:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2984763392 bytes 2024-09-16T11:28:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3006610432 bytes 2024-09-16T11:28:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3008924672 bytes 2024-09-16T11:28:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3033751552 bytes 2024-09-16T11:28:22.891 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:28:22.889+0000 7f48ae988640 -1 reset not still connected to 0x55597c343790 2024-09-16T11:28:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3041208320 bytes 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_8 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:28:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2024-09-16T11:28:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:28:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-16T11:28:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:28:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-16T11:28:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:28:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_8': 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12d419d9bb67 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12d419d9bb67 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:23 2024 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:23 2024 2024-09-16T11:28:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:23 2024 2024-09-16T11:28:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:28:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:28:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 8f02c93f-0a2c-444a-a327-af8d0dcd2b89 2024-09-16T11:28:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:28:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:28:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-16T11:28:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:28:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:28:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=11 2024-09-16T11:28:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:28:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:28:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 49169408 bytes 2024-09-16T11:28:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 49794048 bytes 2024-09-16T11:28:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 51178496 bytes 2024-09-16T11:28:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 77511680 bytes 2024-09-16T11:28:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 122366976 bytes 2024-09-16T11:28:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 127544320 bytes 2024-09-16T11:28:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 216813568 bytes 2024-09-16T11:28:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 217450496 bytes 2024-09-16T11:28:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 223768576 bytes 2024-09-16T11:28:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 293730304 bytes 2024-09-16T11:28:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 326696960 bytes 2024-09-16T11:28:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 390440960 bytes 2024-09-16T11:28:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 434480128 bytes 2024-09-16T11:28:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 453593088 bytes 2024-09-16T11:28:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 558343168 bytes 2024-09-16T11:28:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 598473728 bytes 2024-09-16T11:28:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 639089664 bytes 2024-09-16T11:28:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 639710208 bytes 2024-09-16T11:28:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 724346880 bytes 2024-09-16T11:28:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 772352000 bytes 2024-09-16T11:28:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 826748928 bytes 2024-09-16T11:28:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 850535424 bytes 2024-09-16T11:28:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 919528448 bytes 2024-09-16T11:28:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 938930176 bytes 2024-09-16T11:28:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 962659328 bytes 2024-09-16T11:28:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1009381376 bytes 2024-09-16T11:28:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1085348864 bytes 2024-09-16T11:28:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1126666240 bytes 2024-09-16T11:28:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1173889024 bytes 2024-09-16T11:28:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1200865280 bytes 2024-09-16T11:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1221673984 bytes 2024-09-16T11:28:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1270236160 bytes 2024-09-16T11:28:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1302731776 bytes 2024-09-16T11:28:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1324427264 bytes 2024-09-16T11:28:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1343994880 bytes 2024-09-16T11:28:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1386166272 bytes 2024-09-16T11:28:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1420105728 bytes 2024-09-16T11:28:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1463613440 bytes 2024-09-16T11:28:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1486609408 bytes 2024-09-16T11:28:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1546865664 bytes 2024-09-16T11:28:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1585832960 bytes 2024-09-16T11:28:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1648885760 bytes 2024-09-16T11:28:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1713132544 bytes 2024-09-16T11:28:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1773471744 bytes 2024-09-16T11:28:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1803346944 bytes 2024-09-16T11:28:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1822999552 bytes 2024-09-16T11:28:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1823617024 bytes 2024-09-16T11:28:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1857390592 bytes 2024-09-16T11:28:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1896837120 bytes 2024-09-16T11:28:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1919822848 bytes 2024-09-16T11:28:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1970184192 bytes 2024-09-16T11:28:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1998682112 bytes 2024-09-16T11:28:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2061913088 bytes 2024-09-16T11:28:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2087530496 bytes 2024-09-16T11:28:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2140788736 bytes 2024-09-16T11:28:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2166195200 bytes 2024-09-16T11:28:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2191807488 bytes 2024-09-16T11:28:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2212091904 bytes 2024-09-16T11:28:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2240370688 bytes 2024-09-16T11:28:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2264123392 bytes 2024-09-16T11:28:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2295564288 bytes 2024-09-16T11:28:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2340982784 bytes 2024-09-16T11:28:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2380463104 bytes 2024-09-16T11:28:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2395634688 bytes 2024-09-16T11:28:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2404029440 bytes 2024-09-16T11:28:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2452143104 bytes 2024-09-16T11:28:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2483862528 bytes 2024-09-16T11:28:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2524113920 bytes 2024-09-16T11:28:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2565056512 bytes 2024-09-16T11:28:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2591645696 bytes 2024-09-16T11:28:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2623068160 bytes 2024-09-16T11:28:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2693933056 bytes 2024-09-16T11:28:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2756896768 bytes 2024-09-16T11:28:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2802605056 bytes 2024-09-16T11:28:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 2809072640 bytes 2024-09-16T11:28:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 2914960384 bytes 2024-09-16T11:28:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 2955225088 bytes 2024-09-16T11:28:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:28:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_9 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-16T11:28:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:28:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_9': 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12e0cab85170 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12e0cab85170 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:28:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:35 2024 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:35 2024 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:35 2024 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: ad8cd2a1-83b6-4d15-ba11-daa8d98ddb16 2024-09-16T11:28:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:28:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:28:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-16T11:28:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:28:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:28:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:28:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:28:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:28:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 69426176 bytes 2024-09-16T11:28:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 114710528 bytes 2024-09-16T11:28:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 184991744 bytes 2024-09-16T11:28:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 238017536 bytes 2024-09-16T11:28:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 277758976 bytes 2024-09-16T11:28:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 297622528 bytes 2024-09-16T11:28:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 376219648 bytes 2024-09-16T11:28:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 418169856 bytes 2024-09-16T11:28:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 444858368 bytes 2024-09-16T11:28:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 493394944 bytes 2024-09-16T11:28:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 545939456 bytes 2024-09-16T11:28:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 564167680 bytes 2024-09-16T11:28:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 588515328 bytes 2024-09-16T11:28:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 628165632 bytes 2024-09-16T11:28:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 708961280 bytes 2024-09-16T11:28:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 745576448 bytes 2024-09-16T11:28:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 750162944 bytes 2024-09-16T11:28:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 787312640 bytes 2024-09-16T11:28:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 795607040 bytes 2024-09-16T11:28:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 842195968 bytes 2024-09-16T11:28:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 914360320 bytes 2024-09-16T11:28:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 990785536 bytes 2024-09-16T11:28:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1054073856 bytes 2024-09-16T11:28:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1086203904 bytes 2024-09-16T11:28:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1095942144 bytes 2024-09-16T11:28:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1135710208 bytes 2024-09-16T11:28:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1155273728 bytes 2024-09-16T11:28:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1188186112 bytes 2024-09-16T11:28:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1257137152 bytes 2024-09-16T11:28:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1347085312 bytes 2024-09-16T11:28:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1381122048 bytes 2024-09-16T11:28:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1442127872 bytes 2024-09-16T11:28:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1508062208 bytes 2024-09-16T11:28:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1528749056 bytes 2024-09-16T11:28:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1548624896 bytes 2024-09-16T11:28:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1588554752 bytes 2024-09-16T11:28:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1648036864 bytes 2024-09-16T11:28:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1696351232 bytes 2024-09-16T11:28:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1717460992 bytes 2024-09-16T11:28:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1795245056 bytes 2024-09-16T11:28:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1886168064 bytes 2024-09-16T11:28:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1895757824 bytes 2024-09-16T11:28:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1940940800 bytes 2024-09-16T11:28:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1981316096 bytes 2024-09-16T11:28:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2007017472 bytes 2024-09-16T11:28:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2092391424 bytes 2024-09-16T11:28:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2154715136 bytes 2024-09-16T11:28:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2208358400 bytes 2024-09-16T11:28:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2251765760 bytes 2024-09-16T11:28:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2288990208 bytes 2024-09-16T11:28:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2299968512 bytes 2024-09-16T11:28:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2339673088 bytes 2024-09-16T11:28:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2383191040 bytes 2024-09-16T11:28:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2436232192 bytes 2024-09-16T11:28:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2489670656 bytes 2024-09-16T11:28:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2519202816 bytes 2024-09-16T11:28:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2597494784 bytes 2024-09-16T11:28:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2653453312 bytes 2024-09-16T11:28:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_10 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2024-09-16T11:28:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:28:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-16T11:28:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:28:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-16T11:28:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:28:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_10': 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12eb22e2d4e5 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12eb22e2d4e5 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:28:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:45 2024 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:45 2024 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:45 2024 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c727cb2e-9ff7-4230-b1cb-e1a3d78fbd0e 2024-09-16T11:28:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:28:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:28:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-16T11:28:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:28:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:28:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=8 2024-09-16T11:28:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:28:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:28:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 39426048 bytes 2024-09-16T11:28:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 87804928 bytes 2024-09-16T11:28:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 98955264 bytes 2024-09-16T11:28:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 132059136 bytes 2024-09-16T11:28:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 197228544 bytes 2024-09-16T11:28:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 239173632 bytes 2024-09-16T11:28:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 267379712 bytes 2024-09-16T11:28:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 338584576 bytes 2024-09-16T11:28:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 381983744 bytes 2024-09-16T11:28:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 397415424 bytes 2024-09-16T11:28:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 410047488 bytes 2024-09-16T11:28:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 437565440 bytes 2024-09-16T11:28:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 490732544 bytes 2024-09-16T11:28:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 500826112 bytes 2024-09-16T11:28:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 526914560 bytes 2024-09-16T11:28:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 559775744 bytes 2024-09-16T11:28:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 601187328 bytes 2024-09-16T11:28:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 653565952 bytes 2024-09-16T11:28:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 681054208 bytes 2024-09-16T11:28:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 736102400 bytes 2024-09-16T11:28:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 806479872 bytes 2024-09-16T11:28:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 891256832 bytes 2024-09-16T11:28:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 953885696 bytes 2024-09-16T11:28:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 995528704 bytes 2024-09-16T11:28:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1064700928 bytes 2024-09-16T11:28:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1086844928 bytes 2024-09-16T11:28:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1140013056 bytes 2024-09-16T11:28:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1142722560 bytes 2024-09-16T11:28:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1151776768 bytes 2024-09-16T11:28:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1243252736 bytes 2024-09-16T11:28:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1333700608 bytes 2024-09-16T11:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1370908672 bytes 2024-09-16T11:28:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1388454912 bytes 2024-09-16T11:28:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1537704960 bytes 2024-09-16T11:28:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1559888896 bytes 2024-09-16T11:28:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1580544000 bytes 2024-09-16T11:28:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1660979200 bytes 2024-09-16T11:28:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1706916864 bytes 2024-09-16T11:28:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1753625600 bytes 2024-09-16T11:28:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1809088512 bytes 2024-09-16T11:28:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1809714176 bytes 2024-09-16T11:28:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1825518592 bytes 2024-09-16T11:28:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1916621824 bytes 2024-09-16T11:28:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1940900864 bytes 2024-09-16T11:28:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1982209024 bytes 2024-09-16T11:28:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2028075008 bytes 2024-09-16T11:28:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2101841920 bytes 2024-09-16T11:28:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2138351616 bytes 2024-09-16T11:28:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2202318848 bytes 2024-09-16T11:28:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2256183296 bytes 2024-09-16T11:28:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2259813376 bytes 2024-09-16T11:28:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:28:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:28:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_11 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:28:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:28:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:28:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:28:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2024-09-16T11:28:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:28:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-16T11:28:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:28:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:28:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-16T11:28:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:28:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_11': 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12f75e989e5 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12f75e989e5 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:28:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:54 2024 2024-09-16T11:28:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:54 2024 2024-09-16T11:28:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:54 2024 2024-09-16T11:28:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:28:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:28:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d5fd87f4-333e-453f-8e09-9ee90d26a1c7 2024-09-16T11:28:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:28:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:28:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-16T11:28:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:28:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:28:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:28:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:28:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:28:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:28:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:28:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 58679296 bytes 2024-09-16T11:28:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 102828032 bytes 2024-09-16T11:28:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 145981440 bytes 2024-09-16T11:28:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 201709568 bytes 2024-09-16T11:28:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 227223552 bytes 2024-09-16T11:28:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 286680064 bytes 2024-09-16T11:28:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 354862080 bytes 2024-09-16T11:28:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 367728640 bytes 2024-09-16T11:28:56.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 415390720 bytes 2024-09-16T11:28:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 445960192 bytes 2024-09-16T11:28:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 486538240 bytes 2024-09-16T11:28:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 515824640 bytes 2024-09-16T11:28:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 532360192 bytes 2024-09-16T11:28:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 581150720 bytes 2024-09-16T11:28:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 624188416 bytes 2024-09-16T11:28:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 646205440 bytes 2024-09-16T11:28:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 666396672 bytes 2024-09-16T11:28:57.961 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T11:28:57.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T11:28:57.961 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T11:28:57.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:28:57.959+0000 7f34977f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38217) UID: 1000 2024-09-16T11:28:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 732784640 bytes 2024-09-16T11:28:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 791407616 bytes 2024-09-16T11:28:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 811129856 bytes 2024-09-16T11:28:58.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:28:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 900161536 bytes 2024-09-16T11:28:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 952062976 bytes 2024-09-16T11:28:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 975730688 bytes 2024-09-16T11:28:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1015373824 bytes 2024-09-16T11:28:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1064690688 bytes 2024-09-16T11:28:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1110272000 bytes 2024-09-16T11:28:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1140366336 bytes 2024-09-16T11:28:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1179266048 bytes 2024-09-16T11:28:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1184434176 bytes 2024-09-16T11:28:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1248358400 bytes 2024-09-16T11:28:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1260106752 bytes 2024-09-16T11:29:00.062 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6587582302836739) > weight (0.25) 2024-09-16T11:29:00.062 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T11:29:00.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T11:29:00.063 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T11:29:00.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:29:00.061+0000 7f6dd41f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 43286) UID: 1000 2024-09-16T11:29:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1341062144 bytes 2024-09-16T11:29:00.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:29:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1403644928 bytes 2024-09-16T11:29:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1459420160 bytes 2024-09-16T11:29:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1483452416 bytes 2024-09-16T11:29:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1484069888 bytes 2024-09-16T11:29:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1495248896 bytes 2024-09-16T11:29:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1514118144 bytes 2024-09-16T11:29:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1597009920 bytes 2024-09-16T11:29:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1633666048 bytes 2024-09-16T11:29:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1643718656 bytes 2024-09-16T11:29:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1684988928 bytes 2024-09-16T11:29:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1685583872 bytes 2024-09-16T11:29:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1758116864 bytes 2024-09-16T11:29:01.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1777058816 bytes 2024-09-16T11:29:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1811995648 bytes 2024-09-16T11:29:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1863707648 bytes 2024-09-16T11:29:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1898991616 bytes 2024-09-16T11:29:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1954862080 bytes 2024-09-16T11:29:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2004525056 bytes 2024-09-16T11:29:02.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2032499712 bytes 2024-09-16T11:29:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2077608960 bytes 2024-09-16T11:29:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2161765376 bytes 2024-09-16T11:29:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2197419008 bytes 2024-09-16T11:29:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2203607040 bytes 2024-09-16T11:29:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2221081600 bytes 2024-09-16T11:29:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2302059520 bytes 2024-09-16T11:29:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2326766592 bytes 2024-09-16T11:29:03.962 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T11:29:03.963 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:29:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:29:03.963 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T11:29:03.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T11:29:03.963 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T11:29:03.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T11:29:03.966 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-16T11:29:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2357331968 bytes 2024-09-16T11:29:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2401545216 bytes 2024-09-16T11:29:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2449623040 bytes 2024-09-16T11:29:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2485998592 bytes 2024-09-16T11:29:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2515017728 bytes 2024-09-16T11:29:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2535208960 bytes 2024-09-16T11:29:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2591606784 bytes 2024-09-16T11:29:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2663837696 bytes 2024-09-16T11:29:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2669660160 bytes 2024-09-16T11:29:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2707742720 bytes 2024-09-16T11:29:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2748669952 bytes 2024-09-16T11:29:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2770040832 bytes 2024-09-16T11:29:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2835153920 bytes 2024-09-16T11:29:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2883640320 bytes 2024-09-16T11:29:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2897321984 bytes 2024-09-16T11:29:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2929271808 bytes 2024-09-16T11:29:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 2980577280 bytes 2024-09-16T11:29:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3028013056 bytes 2024-09-16T11:29:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3034122240 bytes 2024-09-16T11:29:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3055767552 bytes 2024-09-16T11:29:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3121539072 bytes 2024-09-16T11:29:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3190789120 bytes 2024-09-16T11:29:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3213945856 bytes 2024-09-16T11:29:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3306436608 bytes 2024-09-16T11:29:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3347512320 bytes 2024-09-16T11:29:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3412471808 bytes 2024-09-16T11:29:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3462117376 bytes 2024-09-16T11:29:08.063 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T11:29:08.064 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:29:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:29:08.064 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T11:29:08.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:29:08.064 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:29:08.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:29:08.069 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-16T11:29:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8600 IOs, 3512793088 bytes 2024-09-16T11:29:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8700 IOs, 3599403008 bytes 2024-09-16T11:29:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8800 IOs, 3637789696 bytes 2024-09-16T11:29:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8900 IOs, 3827411968 bytes 2024-09-16T11:29:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9000 IOs, 3847883776 bytes 2024-09-16T11:29:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9100 IOs, 3856874496 bytes 2024-09-16T11:29:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9200 IOs, 3873683456 bytes 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_12 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:29:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2024-09-16T11:29:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:29:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-16T11:29:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:29:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-16T11:29:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:29:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-16T11:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_12': 2024-09-16T11:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 131153f7382d 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.131153f7382d 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:09 2024 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:09 2024 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:09 2024 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: a7a5a597-f597-46c5-bbc6-a5aec2f981b0 2024-09-16T11:29:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:29:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:29:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-16T11:29:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:29:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:29:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=8 2024-09-16T11:29:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:29:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:29:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 34265088 bytes 2024-09-16T11:29:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 96893952 bytes 2024-09-16T11:29:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 107531264 bytes 2024-09-16T11:29:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 148593664 bytes 2024-09-16T11:29:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 182937600 bytes 2024-09-16T11:29:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 183526400 bytes 2024-09-16T11:29:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 202218496 bytes 2024-09-16T11:29:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 248936448 bytes 2024-09-16T11:29:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 297873408 bytes 2024-09-16T11:29:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 352911360 bytes 2024-09-16T11:29:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 378033152 bytes 2024-09-16T11:29:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 485555200 bytes 2024-09-16T11:29:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 518980608 bytes 2024-09-16T11:29:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 567309312 bytes 2024-09-16T11:29:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 609341440 bytes 2024-09-16T11:29:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 682811392 bytes 2024-09-16T11:29:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 746632192 bytes 2024-09-16T11:29:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 801005568 bytes 2024-09-16T11:29:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 829293568 bytes 2024-09-16T11:29:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 855805952 bytes 2024-09-16T11:29:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 941387776 bytes 2024-09-16T11:29:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 961427456 bytes 2024-09-16T11:29:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 990417920 bytes 2024-09-16T11:29:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1015961600 bytes 2024-09-16T11:29:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1036720128 bytes 2024-09-16T11:29:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1044716544 bytes 2024-09-16T11:29:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1094550528 bytes 2024-09-16T11:29:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1114852352 bytes 2024-09-16T11:29:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1164308480 bytes 2024-09-16T11:29:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1187561472 bytes 2024-09-16T11:29:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1234151424 bytes 2024-09-16T11:29:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1257073664 bytes 2024-09-16T11:29:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1312530432 bytes 2024-09-16T11:29:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1332127744 bytes 2024-09-16T11:29:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1434812416 bytes 2024-09-16T11:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1481869312 bytes 2024-09-16T11:29:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1614590976 bytes 2024-09-16T11:29:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1674747904 bytes 2024-09-16T11:29:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1708195840 bytes 2024-09-16T11:29:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1730894848 bytes 2024-09-16T11:29:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1822538752 bytes 2024-09-16T11:29:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1870339072 bytes 2024-09-16T11:29:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1876767744 bytes 2024-09-16T11:29:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1921873920 bytes 2024-09-16T11:29:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1941149696 bytes 2024-09-16T11:29:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1980119040 bytes 2024-09-16T11:29:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1998976000 bytes 2024-09-16T11:29:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2032230400 bytes 2024-09-16T11:29:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2057680896 bytes 2024-09-16T11:29:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2099783680 bytes 2024-09-16T11:29:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2122353664 bytes 2024-09-16T11:29:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2127634432 bytes 2024-09-16T11:29:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2159659008 bytes 2024-09-16T11:29:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_13 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:29:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:29:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2024-09-16T11:29:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:29:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-16T11:29:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:29:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-16T11:29:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:29:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-16T11:29:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_13': 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 131dd3ed393 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.131dd3ed393 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:18 2024 2024-09-16T11:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:18 2024 2024-09-16T11:29:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:18 2024 2024-09-16T11:29:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:29:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:29:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f5a2a94d-7ff1-4c32-9b14-48a803629566 2024-09-16T11:29:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:29:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:29:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-16T11:29:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:29:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:29:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:29:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:29:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:29:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 70700032 bytes 2024-09-16T11:29:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 79015936 bytes 2024-09-16T11:29:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 107246592 bytes 2024-09-16T11:29:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 107858944 bytes 2024-09-16T11:29:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 124389376 bytes 2024-09-16T11:29:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 181574656 bytes 2024-09-16T11:29:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 263048192 bytes 2024-09-16T11:29:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 297618432 bytes 2024-09-16T11:29:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 360216576 bytes 2024-09-16T11:29:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 482063360 bytes 2024-09-16T11:29:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 530840576 bytes 2024-09-16T11:29:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 556341248 bytes 2024-09-16T11:29:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 581787648 bytes 2024-09-16T11:29:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 602857472 bytes 2024-09-16T11:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 638721024 bytes 2024-09-16T11:29:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 729956352 bytes 2024-09-16T11:29:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 786894848 bytes 2024-09-16T11:29:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 825818112 bytes 2024-09-16T11:29:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 826456064 bytes 2024-09-16T11:29:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 845959168 bytes 2024-09-16T11:29:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 939271168 bytes 2024-09-16T11:29:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 977656832 bytes 2024-09-16T11:29:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 978297856 bytes 2024-09-16T11:29:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 998114304 bytes 2024-09-16T11:29:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1013436416 bytes 2024-09-16T11:29:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1099657216 bytes 2024-09-16T11:29:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1140374528 bytes 2024-09-16T11:29:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1241001984 bytes 2024-09-16T11:29:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1298625536 bytes 2024-09-16T11:29:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1320896512 bytes 2024-09-16T11:29:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1353730048 bytes 2024-09-16T11:29:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1364978688 bytes 2024-09-16T11:29:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1385563136 bytes 2024-09-16T11:29:24.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1386172416 bytes 2024-09-16T11:29:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1440179200 bytes 2024-09-16T11:29:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1476027392 bytes 2024-09-16T11:29:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1548689408 bytes 2024-09-16T11:29:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1627635712 bytes 2024-09-16T11:29:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1688729600 bytes 2024-09-16T11:29:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1709541376 bytes 2024-09-16T11:29:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1722377216 bytes 2024-09-16T11:29:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1770622976 bytes 2024-09-16T11:29:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1795066880 bytes 2024-09-16T11:29:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1828324352 bytes 2024-09-16T11:29:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1902721024 bytes 2024-09-16T11:29:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1997755392 bytes 2024-09-16T11:29:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1998367744 bytes 2024-09-16T11:29:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2030632960 bytes 2024-09-16T11:29:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2061636608 bytes 2024-09-16T11:29:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2096061440 bytes 2024-09-16T11:29:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2115482624 bytes 2024-09-16T11:29:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2136347648 bytes 2024-09-16T11:29:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2169700352 bytes 2024-09-16T11:29:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2205983744 bytes 2024-09-16T11:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2244122624 bytes 2024-09-16T11:29:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2290269184 bytes 2024-09-16T11:29:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2319308800 bytes 2024-09-16T11:29:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2332746752 bytes 2024-09-16T11:29:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2395688960 bytes 2024-09-16T11:29:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2470236160 bytes 2024-09-16T11:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2502632448 bytes 2024-09-16T11:29:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2582780928 bytes 2024-09-16T11:29:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2586073088 bytes 2024-09-16T11:29:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2622500864 bytes 2024-09-16T11:29:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2684240896 bytes 2024-09-16T11:29:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2776233984 bytes 2024-09-16T11:29:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2824216576 bytes 2024-09-16T11:29:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2873913344 bytes 2024-09-16T11:29:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2908739584 bytes 2024-09-16T11:29:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2996092928 bytes 2024-09-16T11:29:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3006385152 bytes 2024-09-16T11:29:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3038289920 bytes 2024-09-16T11:29:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3068199936 bytes 2024-09-16T11:29:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3126864896 bytes 2024-09-16T11:29:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3187996672 bytes 2024-09-16T11:29:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3214018560 bytes 2024-09-16T11:29:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3253810176 bytes 2024-09-16T11:29:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3286330368 bytes 2024-09-16T11:29:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3340933120 bytes 2024-09-16T11:29:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3408433152 bytes 2024-09-16T11:29:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3412373504 bytes 2024-09-16T11:29:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3462555648 bytes 2024-09-16T11:29:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3536402432 bytes 2024-09-16T11:29:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3567142912 bytes 2024-09-16T11:29:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3615050752 bytes 2024-09-16T11:29:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8600 IOs, 3659289600 bytes 2024-09-16T11:29:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8700 IOs, 3664331776 bytes 2024-09-16T11:29:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8800 IOs, 3695755264 bytes 2024-09-16T11:29:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8900 IOs, 3720947712 bytes 2024-09-16T11:29:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:9000 IOs, 3766649856 bytes 2024-09-16T11:29:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:29:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:29:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_14 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2024-09-16T11:29:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:29:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-16T11:29:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:29:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-16T11:29:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:29:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-16T11:29:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_14': 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13292e0ba8cc 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13292e0ba8cc 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:33 2024 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:33 2024 2024-09-16T11:29:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:33 2024 2024-09-16T11:29:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:29:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:29:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b897dc63-3368-46d2-ae8f-58605bd44840 2024-09-16T11:29:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:29:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:29:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-16T11:29:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:29:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:29:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=10 2024-09-16T11:29:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:29:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:29:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 16148480 bytes 2024-09-16T11:29:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 54273024 bytes 2024-09-16T11:29:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 97985536 bytes 2024-09-16T11:29:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 118002688 bytes 2024-09-16T11:29:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 153532416 bytes 2024-09-16T11:29:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 193768448 bytes 2024-09-16T11:29:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 228712448 bytes 2024-09-16T11:29:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 229337088 bytes 2024-09-16T11:29:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 285547520 bytes 2024-09-16T11:29:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 291702784 bytes 2024-09-16T11:29:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 336932864 bytes 2024-09-16T11:29:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 337531904 bytes 2024-09-16T11:29:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 360692736 bytes 2024-09-16T11:29:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 428844032 bytes 2024-09-16T11:29:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 436221952 bytes 2024-09-16T11:29:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 506448896 bytes 2024-09-16T11:29:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 534668288 bytes 2024-09-16T11:29:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 561273856 bytes 2024-09-16T11:29:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 601214976 bytes 2024-09-16T11:29:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 652029952 bytes 2024-09-16T11:29:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 702748672 bytes 2024-09-16T11:29:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 759455744 bytes 2024-09-16T11:29:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 794745856 bytes 2024-09-16T11:29:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 805927936 bytes 2024-09-16T11:29:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 890689536 bytes 2024-09-16T11:29:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 924273664 bytes 2024-09-16T11:29:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 973727744 bytes 2024-09-16T11:29:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 994853888 bytes 2024-09-16T11:29:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1033037824 bytes 2024-09-16T11:29:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1053928448 bytes 2024-09-16T11:29:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1069530112 bytes 2024-09-16T11:29:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1117035520 bytes 2024-09-16T11:29:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1195865088 bytes 2024-09-16T11:29:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1239681024 bytes 2024-09-16T11:29:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1307001856 bytes 2024-09-16T11:29:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1372984320 bytes 2024-09-16T11:29:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1403355136 bytes 2024-09-16T11:29:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1450054656 bytes 2024-09-16T11:29:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1543939072 bytes 2024-09-16T11:29:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1586250752 bytes 2024-09-16T11:29:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1603191808 bytes 2024-09-16T11:29:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1605750784 bytes 2024-09-16T11:29:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1637587968 bytes 2024-09-16T11:29:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1662382080 bytes 2024-09-16T11:29:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1691665408 bytes 2024-09-16T11:29:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1752504320 bytes 2024-09-16T11:29:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1784830976 bytes 2024-09-16T11:29:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1834957824 bytes 2024-09-16T11:29:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1906235392 bytes 2024-09-16T11:29:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1912513536 bytes 2024-09-16T11:29:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1936842752 bytes 2024-09-16T11:29:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1943251968 bytes 2024-09-16T11:29:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 1961412608 bytes 2024-09-16T11:29:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 1998596096 bytes 2024-09-16T11:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2060171264 bytes 2024-09-16T11:29:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2109936640 bytes 2024-09-16T11:29:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2169859072 bytes 2024-09-16T11:29:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2181576704 bytes 2024-09-16T11:29:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2203564032 bytes 2024-09-16T11:29:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2231664640 bytes 2024-09-16T11:29:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2254288896 bytes 2024-09-16T11:29:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2271245312 bytes 2024-09-16T11:29:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2322796544 bytes 2024-09-16T11:29:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2363366400 bytes 2024-09-16T11:29:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2457320448 bytes 2024-09-16T11:29:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2518335488 bytes 2024-09-16T11:29:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2544888832 bytes 2024-09-16T11:29:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2573170688 bytes 2024-09-16T11:29:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:29:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:29:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_15 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2024-09-16T11:29:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:29:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-16T11:29:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:29:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-16T11:29:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:29:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-16T11:29:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_15': 2024-09-16T11:29:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1335fd81b4b8 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1335fd81b4b8 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:44 2024 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:44 2024 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:44 2024 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 8544ced6-19e9-448e-afa3-3117a33e55fb 2024-09-16T11:29:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:29:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:29:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-16T11:29:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:29:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:29:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=8 2024-09-16T11:29:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:29:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:29:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 39563264 bytes 2024-09-16T11:29:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 75743232 bytes 2024-09-16T11:29:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 114765824 bytes 2024-09-16T11:29:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 134611968 bytes 2024-09-16T11:29:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 186764288 bytes 2024-09-16T11:29:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 234991616 bytes 2024-09-16T11:29:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 235593728 bytes 2024-09-16T11:29:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 277039104 bytes 2024-09-16T11:29:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 358149120 bytes 2024-09-16T11:29:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 463510528 bytes 2024-09-16T11:29:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 565696512 bytes 2024-09-16T11:29:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 618092544 bytes 2024-09-16T11:29:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 664820736 bytes 2024-09-16T11:29:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 742627328 bytes 2024-09-16T11:29:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 766625792 bytes 2024-09-16T11:29:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 886514688 bytes 2024-09-16T11:29:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 920371200 bytes 2024-09-16T11:29:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 968526848 bytes 2024-09-16T11:29:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 999366656 bytes 2024-09-16T11:29:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1017033728 bytes 2024-09-16T11:29:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1022242816 bytes 2024-09-16T11:29:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1072175104 bytes 2024-09-16T11:29:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1095621632 bytes 2024-09-16T11:29:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1125022720 bytes 2024-09-16T11:29:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1144827904 bytes 2024-09-16T11:29:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1165656064 bytes 2024-09-16T11:29:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1186384896 bytes 2024-09-16T11:29:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1236941824 bytes 2024-09-16T11:29:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1242138624 bytes 2024-09-16T11:29:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1256924160 bytes 2024-09-16T11:29:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1280738304 bytes 2024-09-16T11:29:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1305442304 bytes 2024-09-16T11:29:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1323868160 bytes 2024-09-16T11:29:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1388429312 bytes 2024-09-16T11:29:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1530015744 bytes 2024-09-16T11:29:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1553469440 bytes 2024-09-16T11:29:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1582588928 bytes 2024-09-16T11:29:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1631130624 bytes 2024-09-16T11:29:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1664940032 bytes 2024-09-16T11:29:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1684829184 bytes 2024-09-16T11:29:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1685436416 bytes 2024-09-16T11:29:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1735132160 bytes 2024-09-16T11:29:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1788860416 bytes 2024-09-16T11:29:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1824903168 bytes 2024-09-16T11:29:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1825482752 bytes 2024-09-16T11:29:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1850460160 bytes 2024-09-16T11:29:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1932364800 bytes 2024-09-16T11:29:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1981762560 bytes 2024-09-16T11:29:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2049004544 bytes 2024-09-16T11:29:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2096092160 bytes 2024-09-16T11:29:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2132557824 bytes 2024-09-16T11:29:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:29:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:29:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_16 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:29:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2024-09-16T11:29:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:29:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-16T11:29:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:29:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:29:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-16T11:29:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:29:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_16': 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1341596ad21b 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1341596ad21b 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:29:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:53 2024 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:53 2024 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:53 2024 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b478e757-78fd-4641-8d2d-50753c43edcb 2024-09-16T11:29:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-16T11:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:29:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:29:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:29:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:29:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:29:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 64573440 bytes 2024-09-16T11:29:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 87757824 bytes 2024-09-16T11:29:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 130061312 bytes 2024-09-16T11:29:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 263960576 bytes 2024-09-16T11:29:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 275898368 bytes 2024-09-16T11:29:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 290028544 bytes 2024-09-16T11:29:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 292215808 bytes 2024-09-16T11:29:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 330637312 bytes 2024-09-16T11:29:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 387055616 bytes 2024-09-16T11:29:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 387672064 bytes 2024-09-16T11:29:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 390635520 bytes 2024-09-16T11:29:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 470372352 bytes 2024-09-16T11:29:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 514230272 bytes 2024-09-16T11:29:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 572740608 bytes 2024-09-16T11:29:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 573362176 bytes 2024-09-16T11:29:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 600113152 bytes 2024-09-16T11:29:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 618345472 bytes 2024-09-16T11:29:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 671643648 bytes 2024-09-16T11:29:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 730756096 bytes 2024-09-16T11:29:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 752505856 bytes 2024-09-16T11:29:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 790746112 bytes 2024-09-16T11:29:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 794125312 bytes 2024-09-16T11:29:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 813705216 bytes 2024-09-16T11:29:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 862477312 bytes 2024-09-16T11:29:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 893884416 bytes 2024-09-16T11:29:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 895375360 bytes 2024-09-16T11:29:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 920094720 bytes 2024-09-16T11:29:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 993335296 bytes 2024-09-16T11:29:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1032320000 bytes 2024-09-16T11:29:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1054173184 bytes 2024-09-16T11:29:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1075132416 bytes 2024-09-16T11:29:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1094463488 bytes 2024-09-16T11:29:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1120053248 bytes 2024-09-16T11:29:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1140772864 bytes 2024-09-16T11:29:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1147738112 bytes 2024-09-16T11:29:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1200069632 bytes 2024-09-16T11:29:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1290156032 bytes 2024-09-16T11:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1333630976 bytes 2024-09-16T11:29:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1393914880 bytes 2024-09-16T11:29:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1446481920 bytes 2024-09-16T11:30:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1468864512 bytes 2024-09-16T11:30:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1487023104 bytes 2024-09-16T11:30:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1527470080 bytes 2024-09-16T11:30:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1601607680 bytes 2024-09-16T11:30:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1640118272 bytes 2024-09-16T11:30:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1671085056 bytes 2024-09-16T11:30:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1730643968 bytes 2024-09-16T11:30:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1752132608 bytes 2024-09-16T11:30:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1814646784 bytes 2024-09-16T11:30:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1815254016 bytes 2024-09-16T11:30:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1862000640 bytes 2024-09-16T11:30:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1916440576 bytes 2024-09-16T11:30:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2004777984 bytes 2024-09-16T11:30:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2037720064 bytes 2024-09-16T11:30:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2111575040 bytes 2024-09-16T11:30:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2171966464 bytes 2024-09-16T11:30:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2214215680 bytes 2024-09-16T11:30:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2232280064 bytes 2024-09-16T11:30:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2270916608 bytes 2024-09-16T11:30:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2287857664 bytes 2024-09-16T11:30:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2320546816 bytes 2024-09-16T11:30:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2378724352 bytes 2024-09-16T11:30:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2416276480 bytes 2024-09-16T11:30:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2455830528 bytes 2024-09-16T11:30:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2478905344 bytes 2024-09-16T11:30:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2509490176 bytes 2024-09-16T11:30:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2541168640 bytes 2024-09-16T11:30:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2546001920 bytes 2024-09-16T11:30:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2573754368 bytes 2024-09-16T11:30:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2618030080 bytes 2024-09-16T11:30:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2628260864 bytes 2024-09-16T11:30:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2698840064 bytes 2024-09-16T11:30:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2764512256 bytes 2024-09-16T11:30:04.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2638117059072883) > weight (0.25) 2024-09-16T11:30:04.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7852815174127054) > weight (0.25) 2024-09-16T11:30:04.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5449357939735372) > weight (0.25) 2024-09-16T11:30:04.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8403682107209098) > weight (0.25) 2024-09-16T11:30:04.967 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-16T11:30:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2765129728 bytes 2024-09-16T11:30:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 2788019200 bytes 2024-09-16T11:30:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 2851957760 bytes 2024-09-16T11:30:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 2886437888 bytes 2024-09-16T11:30:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 2947591168 bytes 2024-09-16T11:30:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3011136512 bytes 2024-09-16T11:30:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3067825152 bytes 2024-09-16T11:30:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3093723136 bytes 2024-09-16T11:30:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_17 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-16T11:30:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-16T11:30:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_17': 2024-09-16T11:30:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 134d1bd8745f 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.134d1bd8745f 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:06 2024 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:06 2024 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:06 2024 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: a73d5ffa-b174-4dd8-82e4-e204e403ef6d 2024-09-16T11:30:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-16T11:30:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:30:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:30:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 26673152 bytes 2024-09-16T11:30:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 64495616 bytes 2024-09-16T11:30:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 144168960 bytes 2024-09-16T11:30:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 195033088 bytes 2024-09-16T11:30:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 275452928 bytes 2024-09-16T11:30:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 316737536 bytes 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 321784832 bytes 2024-09-16T11:30:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 325398528 bytes 2024-09-16T11:30:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 394901504 bytes 2024-09-16T11:30:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 433984512 bytes 2024-09-16T11:30:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 515187712 bytes 2024-09-16T11:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 531685376 bytes 2024-09-16T11:30:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 576904192 bytes 2024-09-16T11:30:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 598700032 bytes 2024-09-16T11:30:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 616570880 bytes 2024-09-16T11:30:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 640203776 bytes 2024-09-16T11:30:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 727836672 bytes 2024-09-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 731362304 bytes 2024-09-16T11:30:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 790094848 bytes 2024-09-16T11:30:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 850076672 bytes 2024-09-16T11:30:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 933440512 bytes 2024-09-16T11:30:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1004537856 bytes 2024-09-16T11:30:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1067991040 bytes 2024-09-16T11:30:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1092343808 bytes 2024-09-16T11:30:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1175228416 bytes 2024-09-16T11:30:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1220637696 bytes 2024-09-16T11:30:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1373686784 bytes 2024-09-16T11:30:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1374300160 bytes 2024-09-16T11:30:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1416093696 bytes 2024-09-16T11:30:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1416711168 bytes 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_18 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-16T11:30:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:30:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2024-09-16T11:30:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-16T11:30:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:30:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-16T11:30:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_18': 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 135836e7027c 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.135836e7027c 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:12 2024 2024-09-16T11:30:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:12 2024 2024-09-16T11:30:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:12 2024 2024-09-16T11:30:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b4ce7a6e-9d23-4e33-892a-2b51e7f4088f 2024-09-16T11:30:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-16T11:30:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:30:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:30:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 34386944 bytes 2024-09-16T11:30:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 65133568 bytes 2024-09-16T11:30:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 112758784 bytes 2024-09-16T11:30:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 144184320 bytes 2024-09-16T11:30:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 187427840 bytes 2024-09-16T11:30:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 278693888 bytes 2024-09-16T11:30:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 366001152 bytes 2024-09-16T11:30:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 424410112 bytes 2024-09-16T11:30:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 440656896 bytes 2024-09-16T11:30:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 495239168 bytes 2024-09-16T11:30:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 547353600 bytes 2024-09-16T11:30:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 580145152 bytes 2024-09-16T11:30:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 598215680 bytes 2024-09-16T11:30:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 679220224 bytes 2024-09-16T11:30:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 734696448 bytes 2024-09-16T11:30:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 765186048 bytes 2024-09-16T11:30:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 841793536 bytes 2024-09-16T11:30:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 853760000 bytes 2024-09-16T11:30:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 884568064 bytes 2024-09-16T11:30:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 949139456 bytes 2024-09-16T11:30:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 991435776 bytes 2024-09-16T11:30:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1057128448 bytes 2024-09-16T11:30:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1113053184 bytes 2024-09-16T11:30:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1146961920 bytes 2024-09-16T11:30:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1211121664 bytes 2024-09-16T11:30:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1262974976 bytes 2024-09-16T11:30:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1269687296 bytes 2024-09-16T11:30:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1370240000 bytes 2024-09-16T11:30:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1416770560 bytes 2024-09-16T11:30:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1463733248 bytes 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_19 2024-09-16T11:30:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2024-09-16T11:30:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-16T11:30:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:30:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-16T11:30:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-16T11:30:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_19': 2024-09-16T11:30:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1364f5f4195e 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1364f5f4195e 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:18 2024 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:18 2024 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:18 2024 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 2d17f5e3-a0dd-42e5-91b8-084a792fce66 2024-09-16T11:30:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-16T11:30:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:30:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=8 2024-09-16T11:30:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 44020736 bytes 2024-09-16T11:30:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 69844992 bytes 2024-09-16T11:30:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 140619776 bytes 2024-09-16T11:30:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 197762048 bytes 2024-09-16T11:30:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 230578176 bytes 2024-09-16T11:30:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 284818432 bytes 2024-09-16T11:30:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 346993664 bytes 2024-09-16T11:30:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 424850432 bytes 2024-09-16T11:30:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 450151424 bytes 2024-09-16T11:30:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 520853504 bytes 2024-09-16T11:30:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 566432768 bytes 2024-09-16T11:30:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 572891136 bytes 2024-09-16T11:30:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 603390976 bytes 2024-09-16T11:30:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 637086720 bytes 2024-09-16T11:30:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 676454400 bytes 2024-09-16T11:30:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 691943424 bytes 2024-09-16T11:30:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 732681216 bytes 2024-09-16T11:30:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 760962048 bytes 2024-09-16T11:30:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 776962048 bytes 2024-09-16T11:30:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 805129216 bytes 2024-09-16T11:30:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 845708288 bytes 2024-09-16T11:30:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 900481024 bytes 2024-09-16T11:30:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 901103616 bytes 2024-09-16T11:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 917108736 bytes 2024-09-16T11:30:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 965554176 bytes 2024-09-16T11:30:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1035655168 bytes 2024-09-16T11:30:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1058589696 bytes 2024-09-16T11:30:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1079834624 bytes 2024-09-16T11:30:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1097659392 bytes 2024-09-16T11:30:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1194364928 bytes 2024-09-16T11:30:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1210731520 bytes 2024-09-16T11:30:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1234250752 bytes 2024-09-16T11:30:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1267828736 bytes 2024-09-16T11:30:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1331870720 bytes 2024-09-16T11:30:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1417244672 bytes 2024-09-16T11:30:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1456352256 bytes 2024-09-16T11:30:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1483237376 bytes 2024-09-16T11:30:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1540456448 bytes 2024-09-16T11:30:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1615424512 bytes 2024-09-16T11:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1667879936 bytes 2024-09-16T11:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1710490624 bytes 2024-09-16T11:30:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1727517696 bytes 2024-09-16T11:30:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1780665344 bytes 2024-09-16T11:30:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1781288960 bytes 2024-09-16T11:30:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1858088960 bytes 2024-09-16T11:30:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1903407104 bytes 2024-09-16T11:30:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2010763264 bytes 2024-09-16T11:30:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2037351424 bytes 2024-09-16T11:30:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2056919040 bytes 2024-09-16T11:30:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2069606400 bytes 2024-09-16T11:30:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2105240576 bytes 2024-09-16T11:30:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2146127872 bytes 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_20 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2024-09-16T11:30:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-16T11:30:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:30:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-16T11:30:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_20': 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 137056e1cc8c 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.137056e1cc8c 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:27 2024 2024-09-16T11:30:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:27 2024 2024-09-16T11:30:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:27 2024 2024-09-16T11:30:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 42e976a5-afcc-4873-8194-2f268da32fe5 2024-09-16T11:30:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-16T11:30:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:30:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:30:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 25368576 bytes 2024-09-16T11:30:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 31266816 bytes 2024-09-16T11:30:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 36335616 bytes 2024-09-16T11:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 118433792 bytes 2024-09-16T11:30:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 203072512 bytes 2024-09-16T11:30:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 232278016 bytes 2024-09-16T11:30:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 241259520 bytes 2024-09-16T11:30:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 299700224 bytes 2024-09-16T11:30:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 304906240 bytes 2024-09-16T11:30:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 321396736 bytes 2024-09-16T11:30:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 372386816 bytes 2024-09-16T11:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 464740352 bytes 2024-09-16T11:30:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 553067520 bytes 2024-09-16T11:30:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 593128448 bytes 2024-09-16T11:30:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 636628992 bytes 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 654194688 bytes 2024-09-16T11:30:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 739979264 bytes 2024-09-16T11:30:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 767217664 bytes 2024-09-16T11:30:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 800311296 bytes 2024-09-16T11:30:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 825382912 bytes 2024-09-16T11:30:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 876065792 bytes 2024-09-16T11:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 904449024 bytes 2024-09-16T11:30:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 982044672 bytes 2024-09-16T11:30:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1032801280 bytes 2024-09-16T11:30:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1081737216 bytes 2024-09-16T11:30:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1086484480 bytes 2024-09-16T11:30:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1092507648 bytes 2024-09-16T11:30:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1187648512 bytes 2024-09-16T11:30:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1206110208 bytes 2024-09-16T11:30:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1227157504 bytes 2024-09-16T11:30:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1262005248 bytes 2024-09-16T11:30:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1343199232 bytes 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_21 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:30:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2024-09-16T11:30:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-16T11:30:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:30:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-16T11:30:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-16T11:30:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_21': 2024-09-16T11:30:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 137cd9fd2be 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.137cd9fd2be 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:33 2024 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:33 2024 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:33 2024 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 24a40005-617d-4202-82db-cfe123cbd527 2024-09-16T11:30:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-16T11:30:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:30:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=11 2024-09-16T11:30:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 28899328 bytes 2024-09-16T11:30:34.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 37315584 bytes 2024-09-16T11:30:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 65991680 bytes 2024-09-16T11:30:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 101990400 bytes 2024-09-16T11:30:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 164637696 bytes 2024-09-16T11:30:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 200589312 bytes 2024-09-16T11:30:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 212664320 bytes 2024-09-16T11:30:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 265671680 bytes 2024-09-16T11:30:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 334675968 bytes 2024-09-16T11:30:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 405109760 bytes 2024-09-16T11:30:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 446453760 bytes 2024-09-16T11:30:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 517936128 bytes 2024-09-16T11:30:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 563847168 bytes 2024-09-16T11:30:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 579132416 bytes 2024-09-16T11:30:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 628817920 bytes 2024-09-16T11:30:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 724868096 bytes 2024-09-16T11:30:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 737953792 bytes 2024-09-16T11:30:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 784422912 bytes 2024-09-16T11:30:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 823857152 bytes 2024-09-16T11:30:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 898276352 bytes 2024-09-16T11:30:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 925277184 bytes 2024-09-16T11:30:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 925872128 bytes 2024-09-16T11:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 926493696 bytes 2024-09-16T11:30:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1000828928 bytes 2024-09-16T11:30:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1029228544 bytes 2024-09-16T11:30:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1058115584 bytes 2024-09-16T11:30:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1093950464 bytes 2024-09-16T11:30:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1181303808 bytes 2024-09-16T11:30:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1198522368 bytes 2024-09-16T11:30:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1267030016 bytes 2024-09-16T11:30:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1310740480 bytes 2024-09-16T11:30:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1341601792 bytes 2024-09-16T11:30:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1391776768 bytes 2024-09-16T11:30:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1432518656 bytes 2024-09-16T11:30:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1459344384 bytes 2024-09-16T11:30:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1525344256 bytes 2024-09-16T11:30:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1559355392 bytes 2024-09-16T11:30:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1626840064 bytes 2024-09-16T11:30:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1646675968 bytes 2024-09-16T11:30:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1724129280 bytes 2024-09-16T11:30:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1759526912 bytes 2024-09-16T11:30:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1801196544 bytes 2024-09-16T11:30:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1875096576 bytes 2024-09-16T11:30:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1927006208 bytes 2024-09-16T11:30:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2013468672 bytes 2024-09-16T11:30:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2056983552 bytes 2024-09-16T11:30:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2093250560 bytes 2024-09-16T11:30:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2188820480 bytes 2024-09-16T11:30:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2259203072 bytes 2024-09-16T11:30:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2349514752 bytes 2024-09-16T11:30:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2398027776 bytes 2024-09-16T11:30:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2425359360 bytes 2024-09-16T11:30:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2473379840 bytes 2024-09-16T11:30:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2518359040 bytes 2024-09-16T11:30:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2549947392 bytes 2024-09-16T11:30:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2596229120 bytes 2024-09-16T11:30:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2605645824 bytes 2024-09-16T11:30:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2714043392 bytes 2024-09-16T11:30:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2722244608 bytes 2024-09-16T11:30:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2755972096 bytes 2024-09-16T11:30:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2776368128 bytes 2024-09-16T11:30:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2823964672 bytes 2024-09-16T11:30:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2833385472 bytes 2024-09-16T11:30:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2925708288 bytes 2024-09-16T11:30:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2973769728 bytes 2024-09-16T11:30:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 3004967936 bytes 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_22 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2024-09-16T11:30:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-16T11:30:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:30:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-16T11:30:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_22': 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13886171fbcb 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13886171fbcb 2024-09-16T11:30:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:45 2024 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:45 2024 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:45 2024 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c33407f3-8da2-4b7c-a242-5b063bcddddb 2024-09-16T11:30:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-16T11:30:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:30:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:30:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 71974912 bytes 2024-09-16T11:30:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 116525056 bytes 2024-09-16T11:30:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 188291072 bytes 2024-09-16T11:30:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 188914688 bytes 2024-09-16T11:30:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 264430592 bytes 2024-09-16T11:30:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 302183424 bytes 2024-09-16T11:30:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 368543744 bytes 2024-09-16T11:30:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 375151616 bytes 2024-09-16T11:30:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 412224512 bytes 2024-09-16T11:30:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 419480576 bytes 2024-09-16T11:30:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 428361728 bytes 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 439992320 bytes 2024-09-16T11:30:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 483769344 bytes 2024-09-16T11:30:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 513132544 bytes 2024-09-16T11:30:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 578409472 bytes 2024-09-16T11:30:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 643776512 bytes 2024-09-16T11:30:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 653713408 bytes 2024-09-16T11:30:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 665971712 bytes 2024-09-16T11:30:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 692131840 bytes 2024-09-16T11:30:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 710759424 bytes 2024-09-16T11:30:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 732475392 bytes 2024-09-16T11:30:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 776458240 bytes 2024-09-16T11:30:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 813328384 bytes 2024-09-16T11:30:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 915969024 bytes 2024-09-16T11:30:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 949732352 bytes 2024-09-16T11:30:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 978327552 bytes 2024-09-16T11:30:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 995103744 bytes 2024-09-16T11:30:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1021159424 bytes 2024-09-16T11:30:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1067094016 bytes 2024-09-16T11:30:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1083530240 bytes 2024-09-16T11:30:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1172093952 bytes 2024-09-16T11:30:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1184842752 bytes 2024-09-16T11:30:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1260563456 bytes 2024-09-16T11:30:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1313052672 bytes 2024-09-16T11:30:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1337240576 bytes 2024-09-16T11:30:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1360925696 bytes 2024-09-16T11:30:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1447031808 bytes 2024-09-16T11:30:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1483223040 bytes 2024-09-16T11:30:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1524272128 bytes 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_23 2024-09-16T11:30:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:30:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2024-09-16T11:30:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:30:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-16T11:30:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:30:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:30:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-16T11:30:53.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:30:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_23': 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13944345dce1 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13944345dce1 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:52 2024 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:52 2024 2024-09-16T11:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:52 2024 2024-09-16T11:30:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:30:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:30:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b442bd0b-01b1-474b-ae4e-23213ee88b71 2024-09-16T11:30:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-16T11:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=10 2024-09-16T11:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:30:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 17030144 bytes 2024-09-16T11:30:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 78401536 bytes 2024-09-16T11:30:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 84302848 bytes 2024-09-16T11:30:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 125513728 bytes 2024-09-16T11:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 149844992 bytes 2024-09-16T11:30:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 150466560 bytes 2024-09-16T11:30:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 170327040 bytes 2024-09-16T11:30:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 216225792 bytes 2024-09-16T11:30:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 257158144 bytes 2024-09-16T11:30:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 300286976 bytes 2024-09-16T11:30:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 329646080 bytes 2024-09-16T11:30:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 391084032 bytes 2024-09-16T11:30:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 480861184 bytes 2024-09-16T11:30:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 481454080 bytes 2024-09-16T11:30:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 507211776 bytes 2024-09-16T11:30:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 574749696 bytes 2024-09-16T11:30:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 646142976 bytes 2024-09-16T11:30:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 692084736 bytes 2024-09-16T11:30:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 735139840 bytes 2024-09-16T11:30:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 780767232 bytes 2024-09-16T11:30:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 796482560 bytes 2024-09-16T11:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 840305664 bytes 2024-09-16T11:30:57.070 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T11:30:57.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T11:30:57.070 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T11:30:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 857355264 bytes 2024-09-16T11:30:57.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:30:57.069+0000 7fb2eb7f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45493) UID: 1000 2024-09-16T11:30:57.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:30:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 949071872 bytes 2024-09-16T11:30:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1002784768 bytes 2024-09-16T11:30:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1071398912 bytes 2024-09-16T11:30:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1101240320 bytes 2024-09-16T11:30:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1166973952 bytes 2024-09-16T11:30:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1227334656 bytes 2024-09-16T11:30:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1269621760 bytes 2024-09-16T11:30:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1297242112 bytes 2024-09-16T11:30:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1314620416 bytes 2024-09-16T11:30:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1388718080 bytes 2024-09-16T11:30:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1443909632 bytes 2024-09-16T11:30:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1497144320 bytes 2024-09-16T11:30:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1513546752 bytes 2024-09-16T11:30:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1602260992 bytes 2024-09-16T11:30:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1632697344 bytes 2024-09-16T11:30:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1675565056 bytes 2024-09-16T11:31:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1718863872 bytes 2024-09-16T11:31:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1800643584 bytes 2024-09-16T11:31:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1839128576 bytes 2024-09-16T11:31:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1857239040 bytes 2024-09-16T11:31:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1930170368 bytes 2024-09-16T11:31:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1975841792 bytes 2024-09-16T11:31:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2019066880 bytes 2024-09-16T11:31:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2061258752 bytes 2024-09-16T11:31:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2086549504 bytes 2024-09-16T11:31:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2098538496 bytes 2024-09-16T11:31:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2161654784 bytes 2024-09-16T11:31:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2186926080 bytes 2024-09-16T11:31:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2187527168 bytes 2024-09-16T11:31:02.071 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T11:31:02.072 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:31:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:31:02.072 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T11:31:02.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:31:02.072 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:31:02.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:31:02.075 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-16T11:31:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2270146560 bytes 2024-09-16T11:31:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2346221568 bytes 2024-09-16T11:31:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2385452032 bytes 2024-09-16T11:31:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2459965440 bytes 2024-09-16T11:31:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2460589056 bytes 2024-09-16T11:31:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2526700544 bytes 2024-09-16T11:31:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2557126656 bytes 2024-09-16T11:31:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2570253312 bytes 2024-09-16T11:31:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2613596160 bytes 2024-09-16T11:31:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:31:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_24 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:31:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2024-09-16T11:31:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:31:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-16T11:31:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:31:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-16T11:31:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:31:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_24': 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13a63a04f6ae 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13a63a04f6ae 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:31:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:03 2024 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:03 2024 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:03 2024 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d4a56ee3-400c-40d4-b957-ce8e33caa539 2024-09-16T11:31:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:31:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:31:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-16T11:31:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:31:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:31:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:31:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:31:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:31:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 57196544 bytes 2024-09-16T11:31:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 61776896 bytes 2024-09-16T11:31:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 76751872 bytes 2024-09-16T11:31:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 101353472 bytes 2024-09-16T11:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 199054336 bytes 2024-09-16T11:31:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 230963200 bytes 2024-09-16T11:31:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 238533632 bytes 2024-09-16T11:31:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 276354048 bytes 2024-09-16T11:31:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 314800128 bytes 2024-09-16T11:31:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 321265664 bytes 2024-09-16T11:31:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 321883136 bytes 2024-09-16T11:31:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 336249856 bytes 2024-09-16T11:31:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 362398720 bytes 2024-09-16T11:31:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 382312448 bytes 2024-09-16T11:31:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 419814400 bytes 2024-09-16T11:31:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 505797632 bytes 2024-09-16T11:31:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 511193088 bytes 2024-09-16T11:31:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 593927168 bytes 2024-09-16T11:31:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 663464960 bytes 2024-09-16T11:31:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 677594112 bytes 2024-09-16T11:31:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 756135936 bytes 2024-09-16T11:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 762797056 bytes 2024-09-16T11:31:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 825350144 bytes 2024-09-16T11:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 873393152 bytes 2024-09-16T11:31:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 900008960 bytes 2024-09-16T11:31:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 938034176 bytes 2024-09-16T11:31:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 952552448 bytes 2024-09-16T11:31:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 975590400 bytes 2024-09-16T11:31:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1003816960 bytes 2024-09-16T11:31:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1091681280 bytes 2024-09-16T11:31:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1127391232 bytes 2024-09-16T11:31:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1177334784 bytes 2024-09-16T11:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1207748608 bytes 2024-09-16T11:31:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1230409728 bytes 2024-09-16T11:31:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1236535296 bytes 2024-09-16T11:31:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1256564736 bytes 2024-09-16T11:31:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1282860032 bytes 2024-09-16T11:31:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1313980416 bytes 2024-09-16T11:31:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1359802368 bytes 2024-09-16T11:31:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1466156032 bytes 2024-09-16T11:31:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1508160512 bytes 2024-09-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1550348288 bytes 2024-09-16T11:31:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1620426752 bytes 2024-09-16T11:31:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1644709888 bytes 2024-09-16T11:31:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1667135488 bytes 2024-09-16T11:31:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1757121536 bytes 2024-09-16T11:31:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1782425600 bytes 2024-09-16T11:31:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1805271040 bytes 2024-09-16T11:31:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1853399040 bytes 2024-09-16T11:31:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1961014272 bytes 2024-09-16T11:31:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1996274688 bytes 2024-09-16T11:31:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2031672320 bytes 2024-09-16T11:31:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2064070656 bytes 2024-09-16T11:31:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2096684032 bytes 2024-09-16T11:31:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2170475520 bytes 2024-09-16T11:31:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2194209792 bytes 2024-09-16T11:31:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2246584320 bytes 2024-09-16T11:31:12.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6592163923727403) > weight (0.25) 2024-09-16T11:31:12.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6571985699877682) > weight (0.25) 2024-09-16T11:31:12.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5311600330075837) > weight (0.25) 2024-09-16T11:31:12.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6562953927361296) > weight (0.25) 2024-09-16T11:31:12.968 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-09-16T11:31:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2321061888 bytes 2024-09-16T11:31:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2322470912 bytes 2024-09-16T11:31:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2364964864 bytes 2024-09-16T11:31:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2410912768 bytes 2024-09-16T11:31:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2462417920 bytes 2024-09-16T11:31:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2548784128 bytes 2024-09-16T11:31:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2552628224 bytes 2024-09-16T11:31:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2598200320 bytes 2024-09-16T11:31:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2732647424 bytes 2024-09-16T11:31:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2757757952 bytes 2024-09-16T11:31:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2774450176 bytes 2024-09-16T11:31:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2802089984 bytes 2024-09-16T11:31:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2823444480 bytes 2024-09-16T11:31:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2858748928 bytes 2024-09-16T11:31:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2898820096 bytes 2024-09-16T11:31:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2920656896 bytes 2024-09-16T11:31:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2993487872 bytes 2024-09-16T11:31:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3034459136 bytes 2024-09-16T11:31:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3079230464 bytes 2024-09-16T11:31:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3147403264 bytes 2024-09-16T11:31:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3204051968 bytes 2024-09-16T11:31:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3213281280 bytes 2024-09-16T11:31:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3229654016 bytes 2024-09-16T11:31:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3263580160 bytes 2024-09-16T11:31:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3321467904 bytes 2024-09-16T11:31:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3382823936 bytes 2024-09-16T11:31:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3424606208 bytes 2024-09-16T11:31:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3467128832 bytes 2024-09-16T11:31:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8600 IOs, 3551390720 bytes 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_25 2024-09-16T11:31:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2024-09-16T11:31:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-16T11:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-16T11:31:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:31:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-16T11:31:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_25': 2024-09-16T11:31:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13b2b43b61f5 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13b2b43b61f5 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:18 2024 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:18 2024 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:18 2024 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:31:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:31:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 72c9a059-c0ba-4ba8-a02d-26f894b5545c 2024-09-16T11:31:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:31:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:31:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-16T11:31:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:31:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:31:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=7 2024-09-16T11:31:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:31:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:31:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 35873792 bytes 2024-09-16T11:31:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 86501376 bytes 2024-09-16T11:31:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 147921920 bytes 2024-09-16T11:31:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 189258752 bytes 2024-09-16T11:31:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 265098240 bytes 2024-09-16T11:31:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 313684992 bytes 2024-09-16T11:31:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 342749184 bytes 2024-09-16T11:31:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 384054272 bytes 2024-09-16T11:31:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 451322880 bytes 2024-09-16T11:31:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 452270080 bytes 2024-09-16T11:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 497265664 bytes 2024-09-16T11:31:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 514592768 bytes 2024-09-16T11:31:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 578444288 bytes 2024-09-16T11:31:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 587947008 bytes 2024-09-16T11:31:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 604502016 bytes 2024-09-16T11:31:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 635266048 bytes 2024-09-16T11:31:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 642634752 bytes 2024-09-16T11:31:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 660150272 bytes 2024-09-16T11:31:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 702308352 bytes 2024-09-16T11:31:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 713214976 bytes 2024-09-16T11:31:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 757383168 bytes 2024-09-16T11:31:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 805904384 bytes 2024-09-16T11:31:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 837043200 bytes 2024-09-16T11:31:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 887809024 bytes 2024-09-16T11:31:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 977425408 bytes 2024-09-16T11:31:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1006069760 bytes 2024-09-16T11:31:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1074278400 bytes 2024-09-16T11:31:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1092325376 bytes 2024-09-16T11:31:23.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1092938752 bytes 2024-09-16T11:31:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1140009984 bytes 2024-09-16T11:31:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1196812288 bytes 2024-09-16T11:31:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1249034240 bytes 2024-09-16T11:31:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1267740672 bytes 2024-09-16T11:31:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1306862592 bytes 2024-09-16T11:31:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1341676544 bytes 2024-09-16T11:31:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1354046464 bytes 2024-09-16T11:31:24.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1388109824 bytes 2024-09-16T11:31:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1388723200 bytes 2024-09-16T11:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1429892096 bytes 2024-09-16T11:31:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1459196928 bytes 2024-09-16T11:31:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1530785792 bytes 2024-09-16T11:31:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1565559808 bytes 2024-09-16T11:31:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1589072896 bytes 2024-09-16T11:31:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1603303424 bytes 2024-09-16T11:31:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1670831104 bytes 2024-09-16T11:31:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1694083072 bytes 2024-09-16T11:31:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1707202560 bytes 2024-09-16T11:31:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:31:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:31:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:31:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_26 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-16T11:31:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-16T11:31:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-16T11:31:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_26': 2024-09-16T11:31:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:31:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:31:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:31:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13be3d382bf0 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13be3d382bf0 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:26 2024 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:26 2024 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:26 2024 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 05dd52ed-bc96-4184-b31a-04d2077723db 2024-09-16T11:31:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:31:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:31:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-16T11:31:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:31:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:31:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:31:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:31:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:31:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 41194496 bytes 2024-09-16T11:31:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 70158336 bytes 2024-09-16T11:31:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 157345792 bytes 2024-09-16T11:31:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 181466112 bytes 2024-09-16T11:31:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 183775232 bytes 2024-09-16T11:31:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 184392704 bytes 2024-09-16T11:31:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 213594112 bytes 2024-09-16T11:31:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 274980864 bytes 2024-09-16T11:31:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 291406848 bytes 2024-09-16T11:31:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 327471104 bytes 2024-09-16T11:31:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 415734784 bytes 2024-09-16T11:31:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 509171712 bytes 2024-09-16T11:31:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 550551552 bytes 2024-09-16T11:31:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 574854144 bytes 2024-09-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 642202624 bytes 2024-09-16T11:31:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 692237312 bytes 2024-09-16T11:31:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 769173504 bytes 2024-09-16T11:31:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 802589696 bytes 2024-09-16T11:31:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 835496960 bytes 2024-09-16T11:31:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 857630720 bytes 2024-09-16T11:31:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 875487232 bytes 2024-09-16T11:31:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 985858048 bytes 2024-09-16T11:31:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1008890880 bytes 2024-09-16T11:31:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1025145856 bytes 2024-09-16T11:31:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1104752640 bytes 2024-09-16T11:31:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1196015616 bytes 2024-09-16T11:31:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1215669248 bytes 2024-09-16T11:31:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1246996480 bytes 2024-09-16T11:31:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1281330176 bytes 2024-09-16T11:31:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1329563648 bytes 2024-09-16T11:31:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1376850944 bytes 2024-09-16T11:31:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1379235840 bytes 2024-09-16T11:31:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1382852608 bytes 2024-09-16T11:31:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1430076416 bytes 2024-09-16T11:31:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1503226880 bytes 2024-09-16T11:31:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1536212992 bytes 2024-09-16T11:31:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1574937600 bytes 2024-09-16T11:31:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1575560192 bytes 2024-09-16T11:31:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1648962560 bytes 2024-09-16T11:31:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1651986432 bytes 2024-09-16T11:31:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1687475200 bytes 2024-09-16T11:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1742531584 bytes 2024-09-16T11:31:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1756796928 bytes 2024-09-16T11:31:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1793864704 bytes 2024-09-16T11:31:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1857938432 bytes 2024-09-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1880085504 bytes 2024-09-16T11:31:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1910449152 bytes 2024-09-16T11:31:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1920791552 bytes 2024-09-16T11:31:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1956527104 bytes 2024-09-16T11:31:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1957152768 bytes 2024-09-16T11:31:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1967905792 bytes 2024-09-16T11:31:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2029077504 bytes 2024-09-16T11:31:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2069073920 bytes 2024-09-16T11:31:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2101843968 bytes 2024-09-16T11:31:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2123445248 bytes 2024-09-16T11:31:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2253378560 bytes 2024-09-16T11:31:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2263533568 bytes 2024-09-16T11:31:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2264157184 bytes 2024-09-16T11:31:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2302208000 bytes 2024-09-16T11:31:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2307686400 bytes 2024-09-16T11:31:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2343640064 bytes 2024-09-16T11:31:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2447457280 bytes 2024-09-16T11:31:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2516608000 bytes 2024-09-16T11:31:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2541236224 bytes 2024-09-16T11:31:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2580452352 bytes 2024-09-16T11:31:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2606124032 bytes 2024-09-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2624420864 bytes 2024-09-16T11:31:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2650195968 bytes 2024-09-16T11:31:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2755244032 bytes 2024-09-16T11:31:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2755868672 bytes 2024-09-16T11:31:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2779834368 bytes 2024-09-16T11:31:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2839112704 bytes 2024-09-16T11:31:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2886977536 bytes 2024-09-16T11:31:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2942005248 bytes 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_27 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:31:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:31:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:31:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:31:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2024-09-16T11:31:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:31:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-16T11:31:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:31:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-16T11:31:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:31:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-16T11:31:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_27': 2024-09-16T11:31:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:31:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13ca3cafa795 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13ca3cafa795 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:39 2024 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:39 2024 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:39 2024 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:31:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:31:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d7cd8c6a-1904-4edd-946f-03b287d93156 2024-09-16T11:31:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:31:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:31:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-16T11:31:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:31:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:31:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:31:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:31:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:31:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 21109760 bytes 2024-09-16T11:31:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 69577728 bytes 2024-09-16T11:31:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 173056000 bytes 2024-09-16T11:31:40.297 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:31:40.293+0000 7f82b9b43640 -1 reset not still connected to 0x55d748497930 2024-09-16T11:31:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 192331776 bytes 2024-09-16T11:31:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 220968960 bytes 2024-09-16T11:31:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 333754368 bytes 2024-09-16T11:31:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 373038080 bytes 2024-09-16T11:31:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 425477120 bytes 2024-09-16T11:31:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 470392832 bytes 2024-09-16T11:31:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 496203776 bytes 2024-09-16T11:31:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 496804864 bytes 2024-09-16T11:31:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 531588096 bytes 2024-09-16T11:31:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 567363584 bytes 2024-09-16T11:31:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 632479744 bytes 2024-09-16T11:31:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 694579200 bytes 2024-09-16T11:31:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 780483584 bytes 2024-09-16T11:31:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 783174656 bytes 2024-09-16T11:31:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 858576896 bytes 2024-09-16T11:31:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 870568960 bytes 2024-09-16T11:31:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 911147008 bytes 2024-09-16T11:31:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1047488512 bytes 2024-09-16T11:31:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1098032128 bytes 2024-09-16T11:31:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1152485376 bytes 2024-09-16T11:31:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1231100928 bytes 2024-09-16T11:31:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1250198528 bytes 2024-09-16T11:31:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1314662400 bytes 2024-09-16T11:31:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1338376192 bytes 2024-09-16T11:31:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1407266816 bytes 2024-09-16T11:31:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1433401344 bytes 2024-09-16T11:31:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1452102656 bytes 2024-09-16T11:31:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1471502336 bytes 2024-09-16T11:31:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1511275520 bytes 2024-09-16T11:31:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1552515072 bytes 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_28 2024-09-16T11:31:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:31:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:31:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:31:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:31:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-16T11:31:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:31:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_28': 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13d614fd57b6 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13d614fd57b6 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:31:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:45 2024 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:45 2024 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:45 2024 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f168dae1-0384-473d-abfb-2c2c4e9f98dc 2024-09-16T11:31:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-16T11:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:31:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:31:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:31:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:31:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 69019648 bytes 2024-09-16T11:31:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 90115072 bytes 2024-09-16T11:31:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 122954752 bytes 2024-09-16T11:31:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 140802048 bytes 2024-09-16T11:31:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 232986624 bytes 2024-09-16T11:31:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 242009088 bytes 2024-09-16T11:31:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 302947328 bytes 2024-09-16T11:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 366093312 bytes 2024-09-16T11:31:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 412800000 bytes 2024-09-16T11:31:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 460250112 bytes 2024-09-16T11:31:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 495448064 bytes 2024-09-16T11:31:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 535971840 bytes 2024-09-16T11:31:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 595139584 bytes 2024-09-16T11:31:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 631136256 bytes 2024-09-16T11:31:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 752649216 bytes 2024-09-16T11:31:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 819036160 bytes 2024-09-16T11:31:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 870220800 bytes 2024-09-16T11:31:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 904194048 bytes 2024-09-16T11:31:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 988305408 bytes 2024-09-16T11:31:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1000488960 bytes 2024-09-16T11:31:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1063165952 bytes 2024-09-16T11:31:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1092780032 bytes 2024-09-16T11:31:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1162439680 bytes 2024-09-16T11:31:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1173676032 bytes 2024-09-16T11:31:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1236379648 bytes 2024-09-16T11:31:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1276255232 bytes 2024-09-16T11:31:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1334549504 bytes 2024-09-16T11:31:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1363145728 bytes 2024-09-16T11:31:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1381378048 bytes 2024-09-16T11:31:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1404149760 bytes 2024-09-16T11:31:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1426653184 bytes 2024-09-16T11:31:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1504377856 bytes 2024-09-16T11:31:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1533351936 bytes 2024-09-16T11:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1621999616 bytes 2024-09-16T11:31:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1708338176 bytes 2024-09-16T11:31:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1768889344 bytes 2024-09-16T11:31:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1802316800 bytes 2024-09-16T11:31:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1802948608 bytes 2024-09-16T11:31:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1837843456 bytes 2024-09-16T11:31:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1893777408 bytes 2024-09-16T11:31:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1941578752 bytes 2024-09-16T11:31:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1982769152 bytes 2024-09-16T11:31:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1995934720 bytes 2024-09-16T11:31:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2050181120 bytes 2024-09-16T11:31:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2079984640 bytes 2024-09-16T11:31:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2133370880 bytes 2024-09-16T11:31:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2205592576 bytes 2024-09-16T11:31:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2255503360 bytes 2024-09-16T11:31:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2335859712 bytes 2024-09-16T11:31:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2372451328 bytes 2024-09-16T11:31:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2422479872 bytes 2024-09-16T11:31:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2439886848 bytes 2024-09-16T11:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2462718976 bytes 2024-09-16T11:31:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2469274624 bytes 2024-09-16T11:31:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2481404928 bytes 2024-09-16T11:31:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2618980352 bytes 2024-09-16T11:31:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2637076480 bytes 2024-09-16T11:31:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2699186176 bytes 2024-09-16T11:31:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2775824384 bytes 2024-09-16T11:31:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2810555392 bytes 2024-09-16T11:31:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2849344512 bytes 2024-09-16T11:31:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2920769536 bytes 2024-09-16T11:31:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2948626432 bytes 2024-09-16T11:31:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 3019467776 bytes 2024-09-16T11:31:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 3041113088 bytes 2024-09-16T11:31:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 3067142144 bytes 2024-09-16T11:31:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 3112459264 bytes 2024-09-16T11:31:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 3113057280 bytes 2024-09-16T11:31:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 3132620800 bytes 2024-09-16T11:31:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3136031744 bytes 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_29 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:31:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:31:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:31:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2024-09-16T11:31:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:31:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-16T11:31:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:31:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:31:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-16T11:31:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:31:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-16T11:31:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_29': 2024-09-16T11:31:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:31:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13e2bab247e 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13e2bab247e 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:58 2024 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:58 2024 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:58 2024 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 28f21500-e6e2-438c-ab9f-f105b1869d6b 2024-09-16T11:31:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:31:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:31:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-16T11:31:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:31:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:31:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:31:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:31:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:31:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:31:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:31:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 10448896 bytes 2024-09-16T11:31:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 26287104 bytes 2024-09-16T11:31:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 68493312 bytes 2024-09-16T11:31:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 69110784 bytes 2024-09-16T11:31:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 147608576 bytes 2024-09-16T11:31:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 155719680 bytes 2024-09-16T11:31:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 158977024 bytes 2024-09-16T11:32:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 194177024 bytes 2024-09-16T11:32:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 237999104 bytes 2024-09-16T11:32:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 315356160 bytes 2024-09-16T11:32:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 373460992 bytes 2024-09-16T11:32:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 405332992 bytes 2024-09-16T11:32:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 440253440 bytes 2024-09-16T11:32:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 481809408 bytes 2024-09-16T11:32:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 518889472 bytes 2024-09-16T11:32:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 524931072 bytes 2024-09-16T11:32:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 538378240 bytes 2024-09-16T11:32:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 596148224 bytes 2024-09-16T11:32:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 672037888 bytes 2024-09-16T11:32:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 718777344 bytes 2024-09-16T11:32:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 730893312 bytes 2024-09-16T11:32:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 786936832 bytes 2024-09-16T11:32:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 883289088 bytes 2024-09-16T11:32:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 944734208 bytes 2024-09-16T11:32:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 975319040 bytes 2024-09-16T11:32:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1040912384 bytes 2024-09-16T11:32:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1138336768 bytes 2024-09-16T11:32:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1158417408 bytes 2024-09-16T11:32:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1199976448 bytes 2024-09-16T11:32:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1238216704 bytes 2024-09-16T11:32:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1326152704 bytes 2024-09-16T11:32:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1348121600 bytes 2024-09-16T11:32:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1415819264 bytes 2024-09-16T11:32:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1491445760 bytes 2024-09-16T11:32:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1538764800 bytes 2024-09-16T11:32:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1614827520 bytes 2024-09-16T11:32:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1630765056 bytes 2024-09-16T11:32:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1651368960 bytes 2024-09-16T11:32:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1670365184 bytes 2024-09-16T11:32:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1703936000 bytes 2024-09-16T11:32:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1758614528 bytes 2024-09-16T11:32:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1781137408 bytes 2024-09-16T11:32:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1819609088 bytes 2024-09-16T11:32:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1863815168 bytes 2024-09-16T11:32:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1871690752 bytes 2024-09-16T11:32:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1919618048 bytes 2024-09-16T11:32:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1954977792 bytes 2024-09-16T11:32:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1983148032 bytes 2024-09-16T11:32:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2010614784 bytes 2024-09-16T11:32:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2071630848 bytes 2024-09-16T11:32:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2098991104 bytes 2024-09-16T11:32:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2140694528 bytes 2024-09-16T11:32:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2233589760 bytes 2024-09-16T11:32:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2257349632 bytes 2024-09-16T11:32:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2275340288 bytes 2024-09-16T11:32:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2339477504 bytes 2024-09-16T11:32:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2443520000 bytes 2024-09-16T11:32:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2454332416 bytes 2024-09-16T11:32:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2497671168 bytes 2024-09-16T11:32:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2516374528 bytes 2024-09-16T11:32:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2569773056 bytes 2024-09-16T11:32:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2621645824 bytes 2024-09-16T11:32:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2632701952 bytes 2024-09-16T11:32:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2689565696 bytes 2024-09-16T11:32:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2773097472 bytes 2024-09-16T11:32:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2812062720 bytes 2024-09-16T11:32:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2829322240 bytes 2024-09-16T11:32:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2896125952 bytes 2024-09-16T11:32:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2941046784 bytes 2024-09-16T11:32:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2973620224 bytes 2024-09-16T11:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3000429568 bytes 2024-09-16T11:32:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3071955968 bytes 2024-09-16T11:32:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3144587264 bytes 2024-09-16T11:32:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3186124800 bytes 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_30 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:32:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-16T11:32:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:32:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_30': 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13ee500c4623 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13ee500c4623 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:32:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:11 2024 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:11 2024 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:11 2024 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d6241982-1b6d-4d65-b374-39b9a477b51a 2024-09-16T11:32:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:32:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:32:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-16T11:32:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:32:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:32:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=10 2024-09-16T11:32:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:32:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:32:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 40963072 bytes 2024-09-16T11:32:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 80718848 bytes 2024-09-16T11:32:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 171564032 bytes 2024-09-16T11:32:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 182969344 bytes 2024-09-16T11:32:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 208072704 bytes 2024-09-16T11:32:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 233034752 bytes 2024-09-16T11:32:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 254394368 bytes 2024-09-16T11:32:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 266387456 bytes 2024-09-16T11:32:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 303400960 bytes 2024-09-16T11:32:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 319270912 bytes 2024-09-16T11:32:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 348583936 bytes 2024-09-16T11:32:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 389783552 bytes 2024-09-16T11:32:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 415620096 bytes 2024-09-16T11:32:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 425400320 bytes 2024-09-16T11:32:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 468740096 bytes 2024-09-16T11:32:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 510917632 bytes 2024-09-16T11:32:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 584876032 bytes 2024-09-16T11:32:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 630938624 bytes 2024-09-16T11:32:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 636497920 bytes 2024-09-16T11:32:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 672244736 bytes 2024-09-16T11:32:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 688728064 bytes 2024-09-16T11:32:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 713409536 bytes 2024-09-16T11:32:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 727052288 bytes 2024-09-16T11:32:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 727656448 bytes 2024-09-16T11:32:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 777196544 bytes 2024-09-16T11:32:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 788155392 bytes 2024-09-16T11:32:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 869957632 bytes 2024-09-16T11:32:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 893259776 bytes 2024-09-16T11:32:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 955322368 bytes 2024-09-16T11:32:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1028662272 bytes 2024-09-16T11:32:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1051892736 bytes 2024-09-16T11:32:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1115597824 bytes 2024-09-16T11:32:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1199918080 bytes 2024-09-16T11:32:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1239215104 bytes 2024-09-16T11:32:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1241651200 bytes 2024-09-16T11:32:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1273390080 bytes 2024-09-16T11:32:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1344203776 bytes 2024-09-16T11:32:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1366063104 bytes 2024-09-16T11:32:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1366701056 bytes 2024-09-16T11:32:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1387267072 bytes 2024-09-16T11:32:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1413999616 bytes 2024-09-16T11:32:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1459906560 bytes 2024-09-16T11:32:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1462786048 bytes 2024-09-16T11:32:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1476430848 bytes 2024-09-16T11:32:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1549249536 bytes 2024-09-16T11:32:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1555633152 bytes 2024-09-16T11:32:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1587079168 bytes 2024-09-16T11:32:19.361 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:32:19.358+0000 7f48ae988640 -1 reset not still connected to 0x55598f011c70 2024-09-16T11:32:19.361 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:32:19.358+0000 7f48ae988640 -1 reset not still connected to 0x555998a33930 2024-09-16T11:32:19.361 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:32:19.358+0000 7f48ae988640 -1 reset not still connected to 0x55599ea0ea90 2024-09-16T11:32:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1633890304 bytes 2024-09-16T11:32:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1667200000 bytes 2024-09-16T11:32:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1702021120 bytes 2024-09-16T11:32:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1752967168 bytes 2024-09-16T11:32:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1781102592 bytes 2024-09-16T11:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 1781686272 bytes 2024-09-16T11:32:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 1815949312 bytes 2024-09-16T11:32:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 1881350144 bytes 2024-09-16T11:32:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 1943350272 bytes 2024-09-16T11:32:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 1988527104 bytes 2024-09-16T11:32:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2012547072 bytes 2024-09-16T11:32:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2046028800 bytes 2024-09-16T11:32:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2047902720 bytes 2024-09-16T11:32:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2106051584 bytes 2024-09-16T11:32:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2157998080 bytes 2024-09-16T11:32:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2200338432 bytes 2024-09-16T11:32:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2240765952 bytes 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_31 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:32:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2024-09-16T11:32:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:32:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-16T11:32:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:32:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-16T11:32:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:32:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_31': 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 13fa861daff5 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.13fa861daff5 2024-09-16T11:32:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:22 2024 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:22 2024 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:22 2024 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 57d96513-857d-403d-8a99-bc4b0fd3e473 2024-09-16T11:32:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-16T11:32:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:32:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:32:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:32:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:32:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:32:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 25836544 bytes 2024-09-16T11:32:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 43590656 bytes 2024-09-16T11:32:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 119114752 bytes 2024-09-16T11:32:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 162714624 bytes 2024-09-16T11:32:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 208478208 bytes 2024-09-16T11:32:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 252376064 bytes 2024-09-16T11:32:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 280124416 bytes 2024-09-16T11:32:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 302935040 bytes 2024-09-16T11:32:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 381364224 bytes 2024-09-16T11:32:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 446141440 bytes 2024-09-16T11:32:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 464976896 bytes 2024-09-16T11:32:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 490235904 bytes 2024-09-16T11:32:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 521225216 bytes 2024-09-16T11:32:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 573569024 bytes 2024-09-16T11:32:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 605841408 bytes 2024-09-16T11:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 606680064 bytes 2024-09-16T11:32:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 633117696 bytes 2024-09-16T11:32:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 639104000 bytes 2024-09-16T11:32:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 748580864 bytes 2024-09-16T11:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 765937664 bytes 2024-09-16T11:32:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 812846080 bytes 2024-09-16T11:32:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 867515392 bytes 2024-09-16T11:32:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 911378432 bytes 2024-09-16T11:32:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 943967232 bytes 2024-09-16T11:32:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1021598720 bytes 2024-09-16T11:32:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1022221312 bytes 2024-09-16T11:32:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1088962560 bytes 2024-09-16T11:32:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1120570368 bytes 2024-09-16T11:32:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1150068736 bytes 2024-09-16T11:32:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1182827520 bytes 2024-09-16T11:32:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1236030464 bytes 2024-09-16T11:32:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1301597184 bytes 2024-09-16T11:32:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1329350656 bytes 2024-09-16T11:32:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1355741184 bytes 2024-09-16T11:32:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1391243264 bytes 2024-09-16T11:32:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1460870144 bytes 2024-09-16T11:32:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1483983872 bytes 2024-09-16T11:32:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1534658560 bytes 2024-09-16T11:32:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1587161088 bytes 2024-09-16T11:32:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1650998272 bytes 2024-09-16T11:32:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1660322816 bytes 2024-09-16T11:32:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1737006080 bytes 2024-09-16T11:32:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1767845888 bytes 2024-09-16T11:32:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1821640704 bytes 2024-09-16T11:32:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1878966272 bytes 2024-09-16T11:32:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1936289792 bytes 2024-09-16T11:32:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1936891904 bytes 2024-09-16T11:32:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1995153408 bytes 2024-09-16T11:32:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2020375552 bytes 2024-09-16T11:32:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2050280448 bytes 2024-09-16T11:32:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2064855040 bytes 2024-09-16T11:32:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2121926656 bytes 2024-09-16T11:32:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2149964800 bytes 2024-09-16T11:32:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2166750208 bytes 2024-09-16T11:32:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2168776704 bytes 2024-09-16T11:32:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2238769152 bytes 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_32 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-16T11:32:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:32:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:32:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:32:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:32:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-16T11:32:33.076 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3077226913721075) > weight (0.25) 2024-09-16T11:32:33.076 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8766599695535676) > weight (0.25) 2024-09-16T11:32:33.076 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.830299624867952) > weight (0.25) 2024-09-16T11:32:33.076 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2878796840916049) > weight (0.25) 2024-09-16T11:32:33.076 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-16T11:32:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:32:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-16T11:32:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_32': 2024-09-16T11:32:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:32:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:32:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:32:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1406a3da1a1d 2024-09-16T11:32:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1406a3da1a1d 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:32 2024 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:32 2024 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:32 2024 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 905fd472-3e29-4634-b6a3-059626393330 2024-09-16T11:32:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:32:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:32:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-16T11:32:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:32:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:32:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:32:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:32:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:32:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 72923136 bytes 2024-09-16T11:32:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 73561088 bytes 2024-09-16T11:32:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 156932096 bytes 2024-09-16T11:32:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 206972928 bytes 2024-09-16T11:32:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 214449152 bytes 2024-09-16T11:32:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 229099520 bytes 2024-09-16T11:32:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 279123968 bytes 2024-09-16T11:32:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 289140736 bytes 2024-09-16T11:32:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 337988608 bytes 2024-09-16T11:32:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 377878528 bytes 2024-09-16T11:32:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 422845440 bytes 2024-09-16T11:32:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 491657216 bytes 2024-09-16T11:32:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 492256256 bytes 2024-09-16T11:32:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 538781696 bytes 2024-09-16T11:32:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 615294976 bytes 2024-09-16T11:32:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 624140288 bytes 2024-09-16T11:32:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 666687488 bytes 2024-09-16T11:32:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 701357056 bytes 2024-09-16T11:32:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 748447744 bytes 2024-09-16T11:32:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 826616832 bytes 2024-09-16T11:32:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 887197696 bytes 2024-09-16T11:32:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 942363648 bytes 2024-09-16T11:32:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1049375744 bytes 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1049986048 bytes 2024-09-16T11:32:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1113077760 bytes 2024-09-16T11:32:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1184377856 bytes 2024-09-16T11:32:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1186770944 bytes 2024-09-16T11:32:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1227377664 bytes 2024-09-16T11:32:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1246053376 bytes 2024-09-16T11:32:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1265424384 bytes 2024-09-16T11:32:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1303776256 bytes 2024-09-16T11:32:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1359546368 bytes 2024-09-16T11:32:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1392574464 bytes 2024-09-16T11:32:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1448720384 bytes 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_33 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:32:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:32:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:32:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2024-09-16T11:32:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:32:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-16T11:32:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:32:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-16T11:32:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:32:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-16T11:32:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_33': 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1412e7997bf2 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1412e7997bf2 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:32:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:39 2024 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:39 2024 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:39 2024 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d843e793-ea22-4d9f-a27b-0c60b2326370 2024-09-16T11:32:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:32:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:32:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-16T11:32:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:32:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:32:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:32:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:32:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:32:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 33456128 bytes 2024-09-16T11:32:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 58317824 bytes 2024-09-16T11:32:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 112276480 bytes 2024-09-16T11:32:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 160321536 bytes 2024-09-16T11:32:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 226702336 bytes 2024-09-16T11:32:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 274235392 bytes 2024-09-16T11:32:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 351805440 bytes 2024-09-16T11:32:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 472583168 bytes 2024-09-16T11:32:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 508538880 bytes 2024-09-16T11:32:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 509144064 bytes 2024-09-16T11:32:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 603440128 bytes 2024-09-16T11:32:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 630965248 bytes 2024-09-16T11:32:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 672730112 bytes 2024-09-16T11:32:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 695609344 bytes 2024-09-16T11:32:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 708841472 bytes 2024-09-16T11:32:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 735299584 bytes 2024-09-16T11:32:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 755577856 bytes 2024-09-16T11:32:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 806003712 bytes 2024-09-16T11:32:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 838318080 bytes 2024-09-16T11:32:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 903180288 bytes 2024-09-16T11:32:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 935062528 bytes 2024-09-16T11:32:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 998565888 bytes 2024-09-16T11:32:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1037446144 bytes 2024-09-16T11:32:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1074395136 bytes 2024-09-16T11:32:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1128511488 bytes 2024-09-16T11:32:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1152842752 bytes 2024-09-16T11:32:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1209869312 bytes 2024-09-16T11:32:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1235909632 bytes 2024-09-16T11:32:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1284487168 bytes 2024-09-16T11:32:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1335056384 bytes 2024-09-16T11:32:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1347574784 bytes 2024-09-16T11:32:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1368292352 bytes 2024-09-16T11:32:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1400258560 bytes 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_34 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:32:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2024-09-16T11:32:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:32:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-16T11:32:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:32:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-16T11:32:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:32:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-16T11:32:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_34': 2024-09-16T11:32:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 141ea1da8707 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.141ea1da8707 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:46 2024 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:46 2024 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:46 2024 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:32:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:32:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 6abecb8f-c955-47d0-8423-8d9b6c3ccf30 2024-09-16T11:32:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:32:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:32:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-16T11:32:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:32:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:32:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=11 2024-09-16T11:32:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:32:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:32:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 100228096 bytes 2024-09-16T11:32:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 107321344 bytes 2024-09-16T11:32:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 130822144 bytes 2024-09-16T11:32:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 197904384 bytes 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 203779072 bytes 2024-09-16T11:32:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 262566912 bytes 2024-09-16T11:32:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 263193600 bytes 2024-09-16T11:32:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 308703232 bytes 2024-09-16T11:32:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 318287872 bytes 2024-09-16T11:32:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 342561792 bytes 2024-09-16T11:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 385359872 bytes 2024-09-16T11:32:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 425192448 bytes 2024-09-16T11:32:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 501105664 bytes 2024-09-16T11:32:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 515490816 bytes 2024-09-16T11:32:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 516092928 bytes 2024-09-16T11:32:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 575645696 bytes 2024-09-16T11:32:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 607740928 bytes 2024-09-16T11:32:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 620463104 bytes 2024-09-16T11:32:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 673607680 bytes 2024-09-16T11:32:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 681475072 bytes 2024-09-16T11:32:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 737229824 bytes 2024-09-16T11:32:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 737848320 bytes 2024-09-16T11:32:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 765776896 bytes 2024-09-16T11:32:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 787633152 bytes 2024-09-16T11:32:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 807365632 bytes 2024-09-16T11:32:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 865273856 bytes 2024-09-16T11:32:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 865903616 bytes 2024-09-16T11:32:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 873146368 bytes 2024-09-16T11:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 950204416 bytes 2024-09-16T11:32:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1005363200 bytes 2024-09-16T11:32:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1065627648 bytes 2024-09-16T11:32:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1179233280 bytes 2024-09-16T11:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1221415936 bytes 2024-09-16T11:32:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1341000704 bytes 2024-09-16T11:32:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1398067200 bytes 2024-09-16T11:32:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1423420416 bytes 2024-09-16T11:32:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1519915008 bytes 2024-09-16T11:32:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1541642240 bytes 2024-09-16T11:32:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1587594240 bytes 2024-09-16T11:32:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1627145216 bytes 2024-09-16T11:32:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1679847424 bytes 2024-09-16T11:32:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1761087488 bytes 2024-09-16T11:32:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1853259776 bytes 2024-09-16T11:32:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1895463936 bytes 2024-09-16T11:32:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1896094720 bytes 2024-09-16T11:32:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1921103872 bytes 2024-09-16T11:32:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1953342464 bytes 2024-09-16T11:32:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1994967040 bytes 2024-09-16T11:32:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2016400384 bytes 2024-09-16T11:32:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2035469312 bytes 2024-09-16T11:32:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2076263424 bytes 2024-09-16T11:32:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2144622592 bytes 2024-09-16T11:32:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2255213568 bytes 2024-09-16T11:32:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2255809536 bytes 2024-09-16T11:32:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2304540672 bytes 2024-09-16T11:32:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2350519296 bytes 2024-09-16T11:32:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2400990208 bytes 2024-09-16T11:32:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2424357888 bytes 2024-09-16T11:32:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2477094912 bytes 2024-09-16T11:32:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2504039424 bytes 2024-09-16T11:32:56.969 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T11:32:56.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T11:32:56.969 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T11:32:56.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:32:56.968+0000 7f431cbf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 47823) UID: 1000 2024-09-16T11:32:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2544546816 bytes 2024-09-16T11:32:57.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:32:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2616941568 bytes 2024-09-16T11:32:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2674975744 bytes 2024-09-16T11:32:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2696632320 bytes 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_35 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:32:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:32:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:32:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:32:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2024-09-16T11:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-16T11:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-16T11:32:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:32:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-16T11:32:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_35': 2024-09-16T11:32:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 142a5b5f7543 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.142a5b5f7543 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:57 2024 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:57 2024 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:57 2024 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:32:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:32:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 22d639e5-b167-4825-bd14-2050baefdab4 2024-09-16T11:32:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:32:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:32:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-16T11:32:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:32:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:32:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:32:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=7 2024-09-16T11:32:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:32:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:32:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 6946816 bytes 2024-09-16T11:32:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 34035712 bytes 2024-09-16T11:32:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 99500032 bytes 2024-09-16T11:32:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 119853056 bytes 2024-09-16T11:32:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 177409024 bytes 2024-09-16T11:32:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 225195008 bytes 2024-09-16T11:32:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 246453248 bytes 2024-09-16T11:32:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 307933184 bytes 2024-09-16T11:32:59.365 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:32:59.363+0000 7f1e20f72640 -1 reset not still connected to 0x55d53ea51520 2024-09-16T11:32:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 364122112 bytes 2024-09-16T11:32:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 373428224 bytes 2024-09-16T11:32:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 419201024 bytes 2024-09-16T11:32:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 439941120 bytes 2024-09-16T11:33:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 485250048 bytes 2024-09-16T11:33:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 544733184 bytes 2024-09-16T11:33:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 574608384 bytes 2024-09-16T11:33:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 617564160 bytes 2024-09-16T11:33:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 623892480 bytes 2024-09-16T11:33:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 664128512 bytes 2024-09-16T11:33:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 730695680 bytes 2024-09-16T11:33:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 761361408 bytes 2024-09-16T11:33:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 830379008 bytes 2024-09-16T11:33:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 865901568 bytes 2024-09-16T11:33:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 896538624 bytes 2024-09-16T11:33:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1002037248 bytes 2024-09-16T11:33:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1016016896 bytes 2024-09-16T11:33:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1073519616 bytes 2024-09-16T11:33:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1085470720 bytes 2024-09-16T11:33:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1164055552 bytes 2024-09-16T11:33:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1192972288 bytes 2024-09-16T11:33:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1223049216 bytes 2024-09-16T11:33:03.287 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:03.286+0000 7f1e20f72640 -1 reset not still connected to 0x55d515f14410 2024-09-16T11:33:03.288 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:03.286+0000 7f1e20f72640 -1 reset not still connected to 0x55d52bc45110 2024-09-16T11:33:03.288 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:03.286+0000 7f1e20f72640 -1 reset not still connected to 0x55d52c64f860 2024-09-16T11:33:03.288 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:03.286+0000 7f1e20f72640 -1 reset not still connected to 0x55d542c97ba0 2024-09-16T11:33:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1248416768 bytes 2024-09-16T11:33:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1292165120 bytes 2024-09-16T11:33:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1343352832 bytes 2024-09-16T11:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1374173184 bytes 2024-09-16T11:33:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1401262080 bytes 2024-09-16T11:33:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1485628416 bytes 2024-09-16T11:33:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1523293184 bytes 2024-09-16T11:33:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1543053312 bytes 2024-09-16T11:33:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1580413952 bytes 2024-09-16T11:33:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1639084032 bytes 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_36 2024-09-16T11:33:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2024-09-16T11:33:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:33:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-16T11:33:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:33:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-16T11:33:05.971 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T11:33:05.971 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:33:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:33:05.972 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T11:33:05.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T11:33:05.972 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T11:33:05.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T11:33:05.976 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-16T11:33:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:33:06.054 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:06.052+0000 7f1e20f72640 -1 reset not still connected to 0x55d52a725d40 2024-09-16T11:33:06.054 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:06.052+0000 7f1e20f72640 -1 reset not still connected to 0x55d52bb6d380 2024-09-16T11:33:06.054 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:33:06.052+0000 7f1e20f72640 -1 reset not still connected to 0x55d52bb6da00 2024-09-16T11:33:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-16T11:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_36': 2024-09-16T11:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1434e0945f39 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1434e0945f39 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:05 2024 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:05 2024 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:05 2024 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 5caa0b20-60a7-472a-a287-3766f979eba6 2024-09-16T11:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:33:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-16T11:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:33:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:33:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:33:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:33:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 86371328 bytes 2024-09-16T11:33:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 130282496 bytes 2024-09-16T11:33:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 175661056 bytes 2024-09-16T11:33:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 256203776 bytes 2024-09-16T11:33:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 281232384 bytes 2024-09-16T11:33:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 296431616 bytes 2024-09-16T11:33:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 336337920 bytes 2024-09-16T11:33:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 374207488 bytes 2024-09-16T11:33:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 386804736 bytes 2024-09-16T11:33:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 432709632 bytes 2024-09-16T11:33:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 510569472 bytes 2024-09-16T11:33:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 527674368 bytes 2024-09-16T11:33:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 569170944 bytes 2024-09-16T11:33:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 600845312 bytes 2024-09-16T11:33:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 643309568 bytes 2024-09-16T11:33:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 667334656 bytes 2024-09-16T11:33:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 671336448 bytes 2024-09-16T11:33:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 671945728 bytes 2024-09-16T11:33:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 705699840 bytes 2024-09-16T11:33:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 706316288 bytes 2024-09-16T11:33:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 724082688 bytes 2024-09-16T11:33:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 751848448 bytes 2024-09-16T11:33:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 788769792 bytes 2024-09-16T11:33:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 834983936 bytes 2024-09-16T11:33:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 860049408 bytes 2024-09-16T11:33:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 911861760 bytes 2024-09-16T11:33:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 951562240 bytes 2024-09-16T11:33:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 952188928 bytes 2024-09-16T11:33:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 989211648 bytes 2024-09-16T11:33:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1021349888 bytes 2024-09-16T11:33:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1102430208 bytes 2024-09-16T11:33:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1136524288 bytes 2024-09-16T11:33:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1204291584 bytes 2024-09-16T11:33:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1229016064 bytes 2024-09-16T11:33:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1272428544 bytes 2024-09-16T11:33:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1321902080 bytes 2024-09-16T11:33:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1387778048 bytes 2024-09-16T11:33:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1449496576 bytes 2024-09-16T11:33:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1515949056 bytes 2024-09-16T11:33:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1532110848 bytes 2024-09-16T11:33:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1603634176 bytes 2024-09-16T11:33:12.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1625435136 bytes 2024-09-16T11:33:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1629276160 bytes 2024-09-16T11:33:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1681128448 bytes 2024-09-16T11:33:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1704766464 bytes 2024-09-16T11:33:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1760778240 bytes 2024-09-16T11:33:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1770037248 bytes 2024-09-16T11:33:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1868898304 bytes 2024-09-16T11:33:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1895942144 bytes 2024-09-16T11:33:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1944338432 bytes 2024-09-16T11:33:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1945823232 bytes 2024-09-16T11:33:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1955399680 bytes 2024-09-16T11:33:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2015625216 bytes 2024-09-16T11:33:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2070894592 bytes 2024-09-16T11:33:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2132756480 bytes 2024-09-16T11:33:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2210615296 bytes 2024-09-16T11:33:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2291890176 bytes 2024-09-16T11:33:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2394529792 bytes 2024-09-16T11:33:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2436109312 bytes 2024-09-16T11:33:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2546386944 bytes 2024-09-16T11:33:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2609497088 bytes 2024-09-16T11:33:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2610110464 bytes 2024-09-16T11:33:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2635084800 bytes 2024-09-16T11:33:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2671720448 bytes 2024-09-16T11:33:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2685619200 bytes 2024-09-16T11:33:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2750912512 bytes 2024-09-16T11:33:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2830896128 bytes 2024-09-16T11:33:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2922694656 bytes 2024-09-16T11:33:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2969879552 bytes 2024-09-16T11:33:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3065397248 bytes 2024-09-16T11:33:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3124319232 bytes 2024-09-16T11:33:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3170612224 bytes 2024-09-16T11:33:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3194226688 bytes 2024-09-16T11:33:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3261904896 bytes 2024-09-16T11:33:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3285709824 bytes 2024-09-16T11:33:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3350341632 bytes 2024-09-16T11:33:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3399413760 bytes 2024-09-16T11:33:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3431386112 bytes 2024-09-16T11:33:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3464327168 bytes 2024-09-16T11:33:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3531790336 bytes 2024-09-16T11:33:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3546711040 bytes 2024-09-16T11:33:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_37 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:33:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:33:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:33:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2024-09-16T11:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-16T11:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:33:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-16T11:33:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:33:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-16T11:33:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_37': 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1447bec3edf1 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1447bec3edf1 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:20 2024 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:20 2024 2024-09-16T11:33:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:20 2024 2024-09-16T11:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: eb944d6a-f8cb-4ae2-a094-e3ad1747c24a 2024-09-16T11:33:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-16T11:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:33:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:33:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:33:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:33:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:33:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 25690112 bytes 2024-09-16T11:33:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 57965568 bytes 2024-09-16T11:33:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 71004160 bytes 2024-09-16T11:33:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 83852288 bytes 2024-09-16T11:33:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 135156736 bytes 2024-09-16T11:33:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 200481792 bytes 2024-09-16T11:33:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 243877888 bytes 2024-09-16T11:33:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 260089856 bytes 2024-09-16T11:33:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 303850496 bytes 2024-09-16T11:33:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 387057664 bytes 2024-09-16T11:33:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 466408448 bytes 2024-09-16T11:33:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 512318464 bytes 2024-09-16T11:33:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 513006592 bytes 2024-09-16T11:33:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 544860160 bytes 2024-09-16T11:33:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 593641472 bytes 2024-09-16T11:33:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 644556800 bytes 2024-09-16T11:33:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 681124864 bytes 2024-09-16T11:33:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 723926016 bytes 2024-09-16T11:33:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 735035392 bytes 2024-09-16T11:33:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 813370368 bytes 2024-09-16T11:33:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 849842176 bytes 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 898542592 bytes 2024-09-16T11:33:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 945639424 bytes 2024-09-16T11:33:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 972764160 bytes 2024-09-16T11:33:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1033690112 bytes 2024-09-16T11:33:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1188513792 bytes 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1251848192 bytes 2024-09-16T11:33:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1303986176 bytes 2024-09-16T11:33:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1325600768 bytes 2024-09-16T11:33:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1346466816 bytes 2024-09-16T11:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1389706240 bytes 2024-09-16T11:33:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1434744832 bytes 2024-09-16T11:33:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1476458496 bytes 2024-09-16T11:33:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1504253952 bytes 2024-09-16T11:33:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1534351360 bytes 2024-09-16T11:33:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1553259520 bytes 2024-09-16T11:33:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1573231616 bytes 2024-09-16T11:33:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1603134464 bytes 2024-09-16T11:33:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1612537856 bytes 2024-09-16T11:33:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1628315648 bytes 2024-09-16T11:33:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1661773824 bytes 2024-09-16T11:33:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1713744896 bytes 2024-09-16T11:33:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1747960832 bytes 2024-09-16T11:33:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1797408768 bytes 2024-09-16T11:33:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1829846016 bytes 2024-09-16T11:33:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1899613184 bytes 2024-09-16T11:33:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2002041856 bytes 2024-09-16T11:33:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2029166592 bytes 2024-09-16T11:33:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2076776448 bytes 2024-09-16T11:33:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2190146560 bytes 2024-09-16T11:33:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2241612800 bytes 2024-09-16T11:33:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2273549312 bytes 2024-09-16T11:33:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2347249664 bytes 2024-09-16T11:33:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2379705344 bytes 2024-09-16T11:33:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2408006656 bytes 2024-09-16T11:33:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2428725248 bytes 2024-09-16T11:33:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2438445056 bytes 2024-09-16T11:33:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2514312192 bytes 2024-09-16T11:33:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2592459776 bytes 2024-09-16T11:33:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2640321536 bytes 2024-09-16T11:33:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2654836736 bytes 2024-09-16T11:33:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2661516288 bytes 2024-09-16T11:33:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2665250816 bytes 2024-09-16T11:33:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2683941888 bytes 2024-09-16T11:33:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2745902080 bytes 2024-09-16T11:33:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2752495616 bytes 2024-09-16T11:33:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2847993856 bytes 2024-09-16T11:33:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2850814976 bytes 2024-09-16T11:33:33.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2908336128 bytes 2024-09-16T11:33:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:33:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:33:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_38 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-16T11:33:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:33:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-16T11:33:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_38': 2024-09-16T11:33:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1450fd6e48a0 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1450fd6e48a0 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:33 2024 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:33 2024 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:33 2024 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:33:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:33:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 7445f68a-3ed3-43b0-b798-83462bd9fca7 2024-09-16T11:33:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:33:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:33:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-16T11:33:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:33:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:33:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:33:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:33:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:33:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 41595904 bytes 2024-09-16T11:33:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 124791808 bytes 2024-09-16T11:33:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 178062336 bytes 2024-09-16T11:33:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 266740736 bytes 2024-09-16T11:33:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 325401600 bytes 2024-09-16T11:33:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 384283648 bytes 2024-09-16T11:33:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 429824000 bytes 2024-09-16T11:33:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 473601024 bytes 2024-09-16T11:33:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 527852544 bytes 2024-09-16T11:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 592702464 bytes 2024-09-16T11:33:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 662528000 bytes 2024-09-16T11:33:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 710768640 bytes 2024-09-16T11:33:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 734027776 bytes 2024-09-16T11:33:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 754364416 bytes 2024-09-16T11:33:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 772630528 bytes 2024-09-16T11:33:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 836429824 bytes 2024-09-16T11:33:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 866309120 bytes 2024-09-16T11:33:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 909015040 bytes 2024-09-16T11:33:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 934740992 bytes 2024-09-16T11:33:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 986223616 bytes 2024-09-16T11:33:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1041500160 bytes 2024-09-16T11:33:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1076670464 bytes 2024-09-16T11:33:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1095544832 bytes 2024-09-16T11:33:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1103210496 bytes 2024-09-16T11:33:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1113340928 bytes 2024-09-16T11:33:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1139163136 bytes 2024-09-16T11:33:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1219190784 bytes 2024-09-16T11:33:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1263677440 bytes 2024-09-16T11:33:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1317042176 bytes 2024-09-16T11:33:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1341620224 bytes 2024-09-16T11:33:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1413472256 bytes 2024-09-16T11:33:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1460109312 bytes 2024-09-16T11:33:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1502796800 bytes 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_39 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:33:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:33:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:33:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:33:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-16T11:33:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:33:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_39': 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 145850998ee6 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.145850998ee6 2024-09-16T11:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:40 2024 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:40 2024 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:40 2024 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: e54108d2-02f3-4f44-b0c9-f084887582f4 2024-09-16T11:33:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:33:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:33:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-16T11:33:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:33:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:33:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:33:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:33:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:33:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 90032128 bytes 2024-09-16T11:33:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 154147840 bytes 2024-09-16T11:33:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 200381440 bytes 2024-09-16T11:33:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 245261312 bytes 2024-09-16T11:33:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 283265024 bytes 2024-09-16T11:33:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 326439936 bytes 2024-09-16T11:33:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 382388224 bytes 2024-09-16T11:33:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 428867584 bytes 2024-09-16T11:33:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 453940224 bytes 2024-09-16T11:33:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 500671488 bytes 2024-09-16T11:33:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 649568256 bytes 2024-09-16T11:33:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 706189312 bytes 2024-09-16T11:33:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 715062272 bytes 2024-09-16T11:33:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 783025152 bytes 2024-09-16T11:33:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 819665920 bytes 2024-09-16T11:33:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 911159296 bytes 2024-09-16T11:33:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 999270400 bytes 2024-09-16T11:33:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 1020523520 bytes 2024-09-16T11:33:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1084001280 bytes 2024-09-16T11:33:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1134891008 bytes 2024-09-16T11:33:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1148815360 bytes 2024-09-16T11:33:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1160938496 bytes 2024-09-16T11:33:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1213143040 bytes 2024-09-16T11:33:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1229290496 bytes 2024-09-16T11:33:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1286175744 bytes 2024-09-16T11:33:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1299001344 bytes 2024-09-16T11:33:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1306925056 bytes 2024-09-16T11:33:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1364922368 bytes 2024-09-16T11:33:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1403701248 bytes 2024-09-16T11:33:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1457552384 bytes 2024-09-16T11:33:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1510110208 bytes 2024-09-16T11:33:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1564162048 bytes 2024-09-16T11:33:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1620363264 bytes 2024-09-16T11:33:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1652593664 bytes 2024-09-16T11:33:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1701977088 bytes 2024-09-16T11:33:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1773596672 bytes 2024-09-16T11:33:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1812175872 bytes 2024-09-16T11:33:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1830343680 bytes 2024-09-16T11:33:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1873342464 bytes 2024-09-16T11:33:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1957272576 bytes 2024-09-16T11:33:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 2003319808 bytes 2024-09-16T11:33:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 2042329088 bytes 2024-09-16T11:33:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2089826304 bytes 2024-09-16T11:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2108494848 bytes 2024-09-16T11:33:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2128790528 bytes 2024-09-16T11:33:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2189128704 bytes 2024-09-16T11:33:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2253826048 bytes 2024-09-16T11:33:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2369835008 bytes 2024-09-16T11:33:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2408426496 bytes 2024-09-16T11:33:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2438464512 bytes 2024-09-16T11:33:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2479647744 bytes 2024-09-16T11:33:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2500658176 bytes 2024-09-16T11:33:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2552754176 bytes 2024-09-16T11:33:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2569966592 bytes 2024-09-16T11:33:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2601134080 bytes 2024-09-16T11:33:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2661399552 bytes 2024-09-16T11:33:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2716958720 bytes 2024-09-16T11:33:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2771566592 bytes 2024-09-16T11:33:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2787823616 bytes 2024-09-16T11:33:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2848324608 bytes 2024-09-16T11:33:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2884886528 bytes 2024-09-16T11:33:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2940524544 bytes 2024-09-16T11:33:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2978794496 bytes 2024-09-16T11:33:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 3032032256 bytes 2024-09-16T11:33:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 3060125696 bytes 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_40 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-16T11:33:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:33:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:33:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:33:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:33:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:33:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2024-09-16T11:33:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:33:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-16T11:33:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:33:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:33:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-16T11:33:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:33:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-16T11:33:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_40': 2024-09-16T11:33:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:33:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 14614fadc6f5 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.14614fadc6f5 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:53 2024 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:53 2024 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:53 2024 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:33:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:33:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: de17e228-9a77-43f1-bfec-27661cda333e 2024-09-16T11:33:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:33:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:33:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-16T11:33:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:33:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:33:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:33:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:33:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:33:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 26353664 bytes 2024-09-16T11:33:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 42728448 bytes 2024-09-16T11:33:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 154968064 bytes 2024-09-16T11:33:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 205541376 bytes 2024-09-16T11:33:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 231681024 bytes 2024-09-16T11:33:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 267192320 bytes 2024-09-16T11:33:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 331800576 bytes 2024-09-16T11:33:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 418253824 bytes 2024-09-16T11:33:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 486669312 bytes 2024-09-16T11:33:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 487294976 bytes 2024-09-16T11:33:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 535717888 bytes 2024-09-16T11:33:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 575194112 bytes 2024-09-16T11:33:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 595923968 bytes 2024-09-16T11:33:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 606102528 bytes 2024-09-16T11:33:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 621215744 bytes 2024-09-16T11:33:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 696146944 bytes 2024-09-16T11:33:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 721199104 bytes 2024-09-16T11:33:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 737808384 bytes 2024-09-16T11:33:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 759316480 bytes 2024-09-16T11:33:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 788023296 bytes 2024-09-16T11:33:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 815884288 bytes 2024-09-16T11:33:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 855715840 bytes 2024-09-16T11:33:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 896701440 bytes 2024-09-16T11:33:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 986358784 bytes 2024-09-16T11:33:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1034366976 bytes 2024-09-16T11:33:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1078853632 bytes 2024-09-16T11:33:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1188370432 bytes 2024-09-16T11:33:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1245638656 bytes 2024-09-16T11:33:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1271146496 bytes 2024-09-16T11:33:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1275248640 bytes 2024-09-16T11:33:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1275869184 bytes 2024-09-16T11:34:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1352068096 bytes 2024-09-16T11:34:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1405236224 bytes 2024-09-16T11:34:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1453154304 bytes 2024-09-16T11:34:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1461529600 bytes 2024-09-16T11:34:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1529021440 bytes 2024-09-16T11:34:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1561536512 bytes 2024-09-16T11:34:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1628677120 bytes 2024-09-16T11:34:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1630722048 bytes 2024-09-16T11:34:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1677363200 bytes 2024-09-16T11:34:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1735444480 bytes 2024-09-16T11:34:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1758153728 bytes 2024-09-16T11:34:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1868490752 bytes 2024-09-16T11:34:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1930906624 bytes 2024-09-16T11:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1986572288 bytes 2024-09-16T11:34:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2089884672 bytes 2024-09-16T11:34:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2137843712 bytes 2024-09-16T11:34:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2164440064 bytes 2024-09-16T11:34:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2185089024 bytes 2024-09-16T11:34:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2259042304 bytes 2024-09-16T11:34:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:34:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:34:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:34:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_41 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:34:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2024-09-16T11:34:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-16T11:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-16T11:34:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:34:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-16T11:34:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_41': 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 146a7f9f2276 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.146a7f9f2276 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:03 2024 2024-09-16T11:34:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:03 2024 2024-09-16T11:34:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:03 2024 2024-09-16T11:34:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:34:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:34:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 316bb7bf-0be4-4d21-b8d9-199ffb446933 2024-09-16T11:34:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-16T11:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:34:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:34:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:34:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:34:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 38716416 bytes 2024-09-16T11:34:04.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 78135296 bytes 2024-09-16T11:34:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 128422912 bytes 2024-09-16T11:34:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 219324416 bytes 2024-09-16T11:34:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 239451136 bytes 2024-09-16T11:34:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 304763904 bytes 2024-09-16T11:34:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 330041344 bytes 2024-09-16T11:34:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 407227392 bytes 2024-09-16T11:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 410895360 bytes 2024-09-16T11:34:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 489223168 bytes 2024-09-16T11:34:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 540724224 bytes 2024-09-16T11:34:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 547257344 bytes 2024-09-16T11:34:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 595580928 bytes 2024-09-16T11:34:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 623450112 bytes 2024-09-16T11:34:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 691631104 bytes 2024-09-16T11:34:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 766942208 bytes 2024-09-16T11:34:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 767558656 bytes 2024-09-16T11:34:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 789476352 bytes 2024-09-16T11:34:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 824326144 bytes 2024-09-16T11:34:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 846586880 bytes 2024-09-16T11:34:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 883700736 bytes 2024-09-16T11:34:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 896533504 bytes 2024-09-16T11:34:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 930470912 bytes 2024-09-16T11:34:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 980613120 bytes 2024-09-16T11:34:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1001926656 bytes 2024-09-16T11:34:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1082284032 bytes 2024-09-16T11:34:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1143990272 bytes 2024-09-16T11:34:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1180189696 bytes 2024-09-16T11:34:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1223120896 bytes 2024-09-16T11:34:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1261283328 bytes 2024-09-16T11:34:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1269323776 bytes 2024-09-16T11:34:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1290557440 bytes 2024-09-16T11:34:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1369278464 bytes 2024-09-16T11:34:10.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1424438272 bytes 2024-09-16T11:34:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1472996352 bytes 2024-09-16T11:34:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1488412672 bytes 2024-09-16T11:34:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1519617024 bytes 2024-09-16T11:34:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1573699584 bytes 2024-09-16T11:34:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1600063488 bytes 2024-09-16T11:34:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1680836608 bytes 2024-09-16T11:34:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1794245632 bytes 2024-09-16T11:34:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1868362752 bytes 2024-09-16T11:34:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1973541888 bytes 2024-09-16T11:34:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1992363008 bytes 2024-09-16T11:34:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2113147904 bytes 2024-09-16T11:34:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2146368512 bytes 2024-09-16T11:34:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2167613440 bytes 2024-09-16T11:34:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2168226816 bytes 2024-09-16T11:34:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2202161152 bytes 2024-09-16T11:34:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2225646592 bytes 2024-09-16T11:34:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2251488256 bytes 2024-09-16T11:34:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2293328896 bytes 2024-09-16T11:34:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2337145856 bytes 2024-09-16T11:34:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2432347136 bytes 2024-09-16T11:34:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2469716992 bytes 2024-09-16T11:34:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2508242944 bytes 2024-09-16T11:34:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2514424832 bytes 2024-09-16T11:34:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2559313920 bytes 2024-09-16T11:34:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2569804800 bytes 2024-09-16T11:34:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2664817664 bytes 2024-09-16T11:34:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2694004736 bytes 2024-09-16T11:34:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2734102528 bytes 2024-09-16T11:34:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2864601088 bytes 2024-09-16T11:34:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2900399104 bytes 2024-09-16T11:34:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2999418880 bytes 2024-09-16T11:34:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 3054258176 bytes 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_42 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:34:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:34:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:34:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2024-09-16T11:34:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:34:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-16T11:34:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:34:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-16T11:34:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:34:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-16T11:34:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_42': 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 147254e192ea 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.147254e192ea 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:16 2024 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:16 2024 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:16 2024 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:34:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:34:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f10dfeca-d8b5-4422-b71a-0f86c6765b14 2024-09-16T11:34:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:34:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:34:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-16T11:34:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:34:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:34:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:34:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:34:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:34:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 15429632 bytes 2024-09-16T11:34:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 67611648 bytes 2024-09-16T11:34:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 68202496 bytes 2024-09-16T11:34:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 117977088 bytes 2024-09-16T11:34:18.346 INFO:tasks.ceph.cluster2.osd.1.smithi103.stderr:2024-09-16T11:34:18.343+0000 7f82b9b43640 -1 reset not still connected to 0x55d75a7d32b0 2024-09-16T11:34:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 170648576 bytes 2024-09-16T11:34:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 187277312 bytes 2024-09-16T11:34:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 247776256 bytes 2024-09-16T11:34:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 352521216 bytes 2024-09-16T11:34:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 408168448 bytes 2024-09-16T11:34:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 504498176 bytes 2024-09-16T11:34:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 535405568 bytes 2024-09-16T11:34:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 646016000 bytes 2024-09-16T11:34:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 720336896 bytes 2024-09-16T11:34:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 772012032 bytes 2024-09-16T11:34:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 852243456 bytes 2024-09-16T11:34:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 902787072 bytes 2024-09-16T11:34:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 969443328 bytes 2024-09-16T11:34:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 1029816320 bytes 2024-09-16T11:34:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1141411840 bytes 2024-09-16T11:34:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1173056512 bytes 2024-09-16T11:34:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1233560576 bytes 2024-09-16T11:34:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1327597568 bytes 2024-09-16T11:34:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1357372416 bytes 2024-09-16T11:34:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1393352704 bytes 2024-09-16T11:34:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1434100736 bytes 2024-09-16T11:34:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1442933760 bytes 2024-09-16T11:34:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1478380544 bytes 2024-09-16T11:34:23.077 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7809723089593119) > weight (0.25) 2024-09-16T11:34:23.078 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7898037400271551) > weight (0.25) 2024-09-16T11:34:23.078 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9479464471752741) > weight (0.25) 2024-09-16T11:34:23.078 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-16T11:34:23.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-16T11:34:23.078 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T11:34:23.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:34:23.078+0000 7fe9d97f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 38177) UID: 1000 2024-09-16T11:34:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1503512576 bytes 2024-09-16T11:34:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1581677568 bytes 2024-09-16T11:34:23.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:34:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1590273024 bytes 2024-09-16T11:34:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1629024256 bytes 2024-09-16T11:34:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1676903424 bytes 2024-09-16T11:34:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1711773696 bytes 2024-09-16T11:34:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1715734528 bytes 2024-09-16T11:34:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1740436480 bytes 2024-09-16T11:34:24.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f8141ff7640 -1 rbd::mirror::ImageReplayer: 0x55f5dd218780 [3/05dd52ed-bc96-4184-b31a-04d2077723db] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f81427f8640 -1 rbd::mirror::ImageReplayer: 0x55f5dd218780 [3/05dd52ed-bc96-4184-b31a-04d2077723db] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f8141ff7640 -1 rbd::mirror::ImageReplayer: 0x55f5dd218280 [3/72c9a059-c0ba-4ba8-a02d-26f894b5545c] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f81427f8640 -1 rbd::mirror::ImageReplayer: 0x55f5dd218280 [3/72c9a059-c0ba-4ba8-a02d-26f894b5545c] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f8141ff7640 -1 rbd::mirror::ImageReplayer: 0x55f5e0c29b80 [3/de17e228-9a77-43f1-bfec-27661cda333e] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f81427f8640 -1 rbd::mirror::ImageReplayer: 0x55f5e0c29b80 [3/de17e228-9a77-43f1-bfec-27661cda333e] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f8141ff7640 -1 rbd::mirror::ImageReplayer: 0x55f5e0810000 [3/8544ced6-19e9-448e-afa3-3117a33e55fb] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.338+0000 7f81427f8640 -1 rbd::mirror::ImageReplayer: 0x55f5e0810000 [3/8544ced6-19e9-448e-afa3-3117a33e55fb] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.339+0000 7f8141ff7640 -1 rbd::mirror::ImageReplayer: 0x55f5e0810280 [3/f168dae1-0384-473d-abfb-2c2c4e9f98dc] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:34:24.339+0000 7f81427f8640 -1 rbd::mirror::ImageReplayer: 0x55f5e0810280 [3/f168dae1-0384-473d-abfb-2c2c4e9f98dc] register_admin_socket_hook: error registering admin socket commands 2024-09-16T11:34:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1741051904 bytes 2024-09-16T11:34:24.494 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:34:24.491+0000 7f1fa11d1640 -1 reset not still connected to 0x565282065860 2024-09-16T11:34:24.494 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:34:24.491+0000 7f1fa11d1640 -1 reset not still connected to 0x5652884b3ba0 2024-09-16T11:34:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1760716800 bytes 2024-09-16T11:34:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1813259264 bytes 2024-09-16T11:34:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1838083072 bytes 2024-09-16T11:34:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1868688384 bytes 2024-09-16T11:34:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1920990208 bytes 2024-09-16T11:34:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1932481536 bytes 2024-09-16T11:34:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2010868736 bytes 2024-09-16T11:34:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2064417792 bytes 2024-09-16T11:34:26.079 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-16T11:34:26.079 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:34:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:34:26.080 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-16T11:34:26.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T11:34:26.080 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T11:34:26.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T11:34:26.083 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-16T11:34:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2141967360 bytes 2024-09-16T11:34:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2157400064 bytes 2024-09-16T11:34:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2180198400 bytes 2024-09-16T11:34:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2194337792 bytes 2024-09-16T11:34:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2227776512 bytes 2024-09-16T11:34:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2285162496 bytes 2024-09-16T11:34:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2288238592 bytes 2024-09-16T11:34:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2312520704 bytes 2024-09-16T11:34:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2384686080 bytes 2024-09-16T11:34:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2399874048 bytes 2024-09-16T11:34:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2432617472 bytes 2024-09-16T11:34:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2491517952 bytes 2024-09-16T11:34:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2618831872 bytes 2024-09-16T11:34:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2648098816 bytes 2024-09-16T11:34:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2713289728 bytes 2024-09-16T11:34:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2772894720 bytes 2024-09-16T11:34:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2787305472 bytes 2024-09-16T11:34:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2814045184 bytes 2024-09-16T11:34:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2849623040 bytes 2024-09-16T11:34:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2866012160 bytes 2024-09-16T11:34:29.298 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:34:29.296+0000 7f1e20f72640 -1 reset not still connected to 0x55d51ba75d40 2024-09-16T11:34:29.299 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:34:29.297+0000 7f1e20f72640 -1 reset not still connected to 0x55d52a725e10 2024-09-16T11:34:29.300 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:34:29.298+0000 7f1e20f72640 -1 reset not still connected to 0x55d5347df6c0 2024-09-16T11:34:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2903196672 bytes 2024-09-16T11:34:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2960249856 bytes 2024-09-16T11:34:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2999238656 bytes 2024-09-16T11:34:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 3047525376 bytes 2024-09-16T11:34:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 3080426496 bytes 2024-09-16T11:34:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 3169570816 bytes 2024-09-16T11:34:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3186879488 bytes 2024-09-16T11:34:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3227758592 bytes 2024-09-16T11:34:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3330978816 bytes 2024-09-16T11:34:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3348947968 bytes 2024-09-16T11:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3409971200 bytes 2024-09-16T11:34:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3415752704 bytes 2024-09-16T11:34:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3443386368 bytes 2024-09-16T11:34:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3450693632 bytes 2024-09-16T11:34:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:34:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:34:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:34:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:34:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:34:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_43 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2024-09-16T11:34:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:34:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-16T11:34:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:34:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-16T11:34:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:34:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-16T11:34:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_43': 2024-09-16T11:34:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:34:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:34:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:34:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1480a27d0ce8 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1480a27d0ce8 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:31 2024 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:31 2024 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:31 2024 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 6eadc152-f19b-4a5d-abbd-4c03a784ef9f 2024-09-16T11:34:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:34:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:34:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-16T11:34:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:34:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:34:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:34:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:34:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:34:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 55046144 bytes 2024-09-16T11:34:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 121078784 bytes 2024-09-16T11:34:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 156921856 bytes 2024-09-16T11:34:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 210756608 bytes 2024-09-16T11:34:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 223362048 bytes 2024-09-16T11:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 302329856 bytes 2024-09-16T11:34:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 322216960 bytes 2024-09-16T11:34:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 396125184 bytes 2024-09-16T11:34:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 434629632 bytes 2024-09-16T11:34:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 477679616 bytes 2024-09-16T11:34:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 497544192 bytes 2024-09-16T11:34:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 580289536 bytes 2024-09-16T11:34:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 606235648 bytes 2024-09-16T11:34:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 649302016 bytes 2024-09-16T11:34:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 671393792 bytes 2024-09-16T11:34:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 796328960 bytes 2024-09-16T11:34:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 916919296 bytes 2024-09-16T11:34:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 936751104 bytes 2024-09-16T11:34:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 973817856 bytes 2024-09-16T11:34:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1010487296 bytes 2024-09-16T11:34:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1051247616 bytes 2024-09-16T11:34:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1119196160 bytes 2024-09-16T11:34:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1192176640 bytes 2024-09-16T11:34:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1201738752 bytes 2024-09-16T11:34:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1209830400 bytes 2024-09-16T11:34:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1243663360 bytes 2024-09-16T11:34:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1304216576 bytes 2024-09-16T11:34:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1360171008 bytes 2024-09-16T11:34:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1364050944 bytes 2024-09-16T11:34:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1364645888 bytes 2024-09-16T11:34:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1444402176 bytes 2024-09-16T11:34:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1465259008 bytes 2024-09-16T11:34:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1510224896 bytes 2024-09-16T11:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1549751296 bytes 2024-09-16T11:34:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1599878144 bytes 2024-09-16T11:34:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1653340160 bytes 2024-09-16T11:34:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1708428288 bytes 2024-09-16T11:34:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1730926592 bytes 2024-09-16T11:34:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1742520320 bytes 2024-09-16T11:34:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1767367680 bytes 2024-09-16T11:34:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1794000896 bytes 2024-09-16T11:34:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1794577408 bytes 2024-09-16T11:34:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1875540992 bytes 2024-09-16T11:34:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1885351936 bytes 2024-09-16T11:34:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1945762816 bytes 2024-09-16T11:34:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2039454720 bytes 2024-09-16T11:34:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2049518592 bytes 2024-09-16T11:34:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2137666560 bytes 2024-09-16T11:34:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2184355840 bytes 2024-09-16T11:34:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2220367872 bytes 2024-09-16T11:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_44 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2024-09-16T11:34:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:34:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-16T11:34:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:34:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-16T11:34:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:34:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-16T11:34:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_44': 2024-09-16T11:34:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:34:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:34:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:34:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 148bac609db3 2024-09-16T11:34:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.148bac609db3 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:41 2024 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:41 2024 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:41 2024 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 0aab3740-d202-4c69-8377-5da1f823faf0 2024-09-16T11:34:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:34:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:34:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-16T11:34:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:34:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:34:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:34:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:34:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:34:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 38396928 bytes 2024-09-16T11:34:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 98394112 bytes 2024-09-16T11:34:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 150298624 bytes 2024-09-16T11:34:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 206042112 bytes 2024-09-16T11:34:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 310196224 bytes 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 375702528 bytes 2024-09-16T11:34:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 421785600 bytes 2024-09-16T11:34:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 474005504 bytes 2024-09-16T11:34:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 495887360 bytes 2024-09-16T11:34:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 561271808 bytes 2024-09-16T11:34:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 647710720 bytes 2024-09-16T11:34:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 674147328 bytes 2024-09-16T11:34:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 689672192 bytes 2024-09-16T11:34:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 713314304 bytes 2024-09-16T11:34:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 735401984 bytes 2024-09-16T11:34:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 758963200 bytes 2024-09-16T11:34:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 830779392 bytes 2024-09-16T11:34:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 833366016 bytes 2024-09-16T11:34:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 851895296 bytes 2024-09-16T11:34:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 995042304 bytes 2024-09-16T11:34:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1016941568 bytes 2024-09-16T11:34:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1076027392 bytes 2024-09-16T11:34:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1095150592 bytes 2024-09-16T11:34:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1121626112 bytes 2024-09-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1189838848 bytes 2024-09-16T11:34:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1289165824 bytes 2024-09-16T11:34:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1348825088 bytes 2024-09-16T11:34:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1376270336 bytes 2024-09-16T11:34:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1381248000 bytes 2024-09-16T11:34:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1398408192 bytes 2024-09-16T11:34:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1437714432 bytes 2024-09-16T11:34:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1481524224 bytes 2024-09-16T11:34:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1518148608 bytes 2024-09-16T11:34:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1532946432 bytes 2024-09-16T11:34:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1652170752 bytes 2024-09-16T11:34:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1723232256 bytes 2024-09-16T11:34:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1762406400 bytes 2024-09-16T11:34:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1795954688 bytes 2024-09-16T11:34:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1852583936 bytes 2024-09-16T11:34:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1873570816 bytes 2024-09-16T11:34:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1900096512 bytes 2024-09-16T11:34:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1945687040 bytes 2024-09-16T11:34:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2009433088 bytes 2024-09-16T11:34:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2048288768 bytes 2024-09-16T11:34:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2089380864 bytes 2024-09-16T11:34:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2138645504 bytes 2024-09-16T11:34:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2139281408 bytes 2024-09-16T11:34:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2197434368 bytes 2024-09-16T11:34:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2235079680 bytes 2024-09-16T11:34:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2257521664 bytes 2024-09-16T11:34:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2298110976 bytes 2024-09-16T11:34:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2348844032 bytes 2024-09-16T11:34:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2358519808 bytes 2024-09-16T11:34:51.976 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.49268804440379244) > weight (0.25) 2024-09-16T11:34:51.976 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41287519622488555) > weight (0.25) 2024-09-16T11:34:51.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6790019187759656) > weight (0.25) 2024-09-16T11:34:51.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5089983974573251) > weight (0.25) 2024-09-16T11:34:51.977 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-16T11:34:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2487633920 bytes 2024-09-16T11:34:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2531760128 bytes 2024-09-16T11:34:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2541709312 bytes 2024-09-16T11:34:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2561939456 bytes 2024-09-16T11:34:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2591319040 bytes 2024-09-16T11:34:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2634798080 bytes 2024-09-16T11:34:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2684598272 bytes 2024-09-16T11:34:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2706394112 bytes 2024-09-16T11:34:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2757066752 bytes 2024-09-16T11:34:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2812469248 bytes 2024-09-16T11:34:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2841735168 bytes 2024-09-16T11:34:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2912264192 bytes 2024-09-16T11:34:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2912884736 bytes 2024-09-16T11:34:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2941589504 bytes 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_45 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2024-09-16T11:34:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:34:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-16T11:34:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:34:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:34:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-16T11:34:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:34:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-16T11:34:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_45': 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 149496a9f8b1 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.149496a9f8b1 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:54 2024 2024-09-16T11:34:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:54 2024 2024-09-16T11:34:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:54 2024 2024-09-16T11:34:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:34:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:34:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 4de1128c-eeed-43f2-8404-66dfce23f601 2024-09-16T11:34:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:34:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:34:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-16T11:34:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:34:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:34:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:34:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:34:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=8 2024-09-16T11:34:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:34:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:34:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 49208320 bytes 2024-09-16T11:34:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 105951232 bytes 2024-09-16T11:34:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 133362688 bytes 2024-09-16T11:34:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 139168768 bytes 2024-09-16T11:34:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 159643648 bytes 2024-09-16T11:34:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 200502272 bytes 2024-09-16T11:34:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 265274368 bytes 2024-09-16T11:34:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 265904128 bytes 2024-09-16T11:34:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 352834560 bytes 2024-09-16T11:34:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 408685568 bytes 2024-09-16T11:34:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 482587648 bytes 2024-09-16T11:34:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 506290176 bytes 2024-09-16T11:34:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 532346880 bytes 2024-09-16T11:34:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 570735616 bytes 2024-09-16T11:34:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 620179456 bytes 2024-09-16T11:34:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 670232576 bytes 2024-09-16T11:34:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 769922048 bytes 2024-09-16T11:34:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 801403904 bytes 2024-09-16T11:34:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 801987584 bytes 2024-09-16T11:34:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 861851648 bytes 2024-09-16T11:34:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 911158272 bytes 2024-09-16T11:34:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 941157376 bytes 2024-09-16T11:34:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1012636672 bytes 2024-09-16T11:34:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1095578624 bytes 2024-09-16T11:34:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1105134592 bytes 2024-09-16T11:34:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1105756160 bytes 2024-09-16T11:35:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1228151808 bytes 2024-09-16T11:35:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1277032448 bytes 2024-09-16T11:35:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1339502592 bytes 2024-09-16T11:35:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1385951232 bytes 2024-09-16T11:35:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1426479104 bytes 2024-09-16T11:35:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1459460096 bytes 2024-09-16T11:35:01.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1492462592 bytes 2024-09-16T11:35:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1541129216 bytes 2024-09-16T11:35:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1626920960 bytes 2024-09-16T11:35:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1643448320 bytes 2024-09-16T11:35:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1690271744 bytes 2024-09-16T11:35:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1740269568 bytes 2024-09-16T11:35:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1765156864 bytes 2024-09-16T11:35:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1796451328 bytes 2024-09-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1826629632 bytes 2024-09-16T11:35:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1905547264 bytes 2024-09-16T11:35:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1937611776 bytes 2024-09-16T11:35:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2012395520 bytes 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_46 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:35:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:35:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:35:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2024-09-16T11:35:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:35:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-16T11:35:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-16T11:35:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:35:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_46': 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 149c2f40352d 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.149c2f40352d 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:35:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:03 2024 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:03 2024 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:03 2024 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 541cb1e3-d977-4c95-ba41-c949127ff0cd 2024-09-16T11:35:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:35:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:35:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-16T11:35:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:35:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:35:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=10 2024-09-16T11:35:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:35:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:35:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 22140928 bytes 2024-09-16T11:35:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 104752128 bytes 2024-09-16T11:35:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 135285760 bytes 2024-09-16T11:35:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 168091648 bytes 2024-09-16T11:35:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 208119808 bytes 2024-09-16T11:35:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 228925440 bytes 2024-09-16T11:35:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 317719552 bytes 2024-09-16T11:35:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 336805888 bytes 2024-09-16T11:35:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 407535616 bytes 2024-09-16T11:35:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 481540096 bytes 2024-09-16T11:35:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 518202368 bytes 2024-09-16T11:35:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 582616064 bytes 2024-09-16T11:35:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 597456896 bytes 2024-09-16T11:35:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 598066176 bytes 2024-09-16T11:35:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 648131584 bytes 2024-09-16T11:35:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 735035392 bytes 2024-09-16T11:35:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 794850304 bytes 2024-09-16T11:35:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 795460608 bytes 2024-09-16T11:35:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 845458432 bytes 2024-09-16T11:35:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 939285504 bytes 2024-09-16T11:35:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 990902272 bytes 2024-09-16T11:35:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1052955648 bytes 2024-09-16T11:35:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1097521152 bytes 2024-09-16T11:35:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1174804480 bytes 2024-09-16T11:35:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1228874752 bytes 2024-09-16T11:35:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1250483200 bytes 2024-09-16T11:35:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1287345152 bytes 2024-09-16T11:35:09.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1397798912 bytes 2024-09-16T11:35:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1411961856 bytes 2024-09-16T11:35:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1468393472 bytes 2024-09-16T11:35:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1553806336 bytes 2024-09-16T11:35:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1597122560 bytes 2024-09-16T11:35:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1705565184 bytes 2024-09-16T11:35:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1746864128 bytes 2024-09-16T11:35:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1786011648 bytes 2024-09-16T11:35:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1811040256 bytes 2024-09-16T11:35:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1811642368 bytes 2024-09-16T11:35:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1866362880 bytes 2024-09-16T11:35:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1866972160 bytes 2024-09-16T11:35:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1910343680 bytes 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1922048000 bytes 2024-09-16T11:35:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1947231232 bytes 2024-09-16T11:35:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2029676544 bytes 2024-09-16T11:35:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2090114048 bytes 2024-09-16T11:35:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2125047808 bytes 2024-09-16T11:35:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2153234432 bytes 2024-09-16T11:35:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2169371648 bytes 2024-09-16T11:35:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2198916096 bytes 2024-09-16T11:35:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2253492224 bytes 2024-09-16T11:35:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2275609600 bytes 2024-09-16T11:35:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2302333952 bytes 2024-09-16T11:35:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2317159424 bytes 2024-09-16T11:35:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2346089472 bytes 2024-09-16T11:35:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2396379136 bytes 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_47 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:35:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:35:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:35:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2024-09-16T11:35:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:35:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-16T11:35:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:35:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-16T11:35:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:35:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_47': 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 14a558a80ccb 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.14a558a80ccb 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:14 2024 2024-09-16T11:35:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:14 2024 2024-09-16T11:35:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:14 2024 2024-09-16T11:35:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:35:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:35:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: a52e8873-193f-487c-b67c-f482a83709db 2024-09-16T11:35:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:35:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:35:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-16T11:35:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:35:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:35:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:35:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:35:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:35:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 649216 bytes 2024-09-16T11:35:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 64143360 bytes 2024-09-16T11:35:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 89428992 bytes 2024-09-16T11:35:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 132684800 bytes 2024-09-16T11:35:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 142194688 bytes 2024-09-16T11:35:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 232717312 bytes 2024-09-16T11:35:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 251256832 bytes 2024-09-16T11:35:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 291988480 bytes 2024-09-16T11:35:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 321558528 bytes 2024-09-16T11:35:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 368209920 bytes 2024-09-16T11:35:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 368790528 bytes 2024-09-16T11:35:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 408543232 bytes 2024-09-16T11:35:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 409149440 bytes 2024-09-16T11:35:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 470038528 bytes 2024-09-16T11:35:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 501444608 bytes 2024-09-16T11:35:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 521884672 bytes 2024-09-16T11:35:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 522517504 bytes 2024-09-16T11:35:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 562037760 bytes 2024-09-16T11:35:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 578659328 bytes 2024-09-16T11:35:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 612758528 bytes 2024-09-16T11:35:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 624844800 bytes 2024-09-16T11:35:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 651404288 bytes 2024-09-16T11:35:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 699691008 bytes 2024-09-16T11:35:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 731856896 bytes 2024-09-16T11:35:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 734791680 bytes 2024-09-16T11:35:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 751963136 bytes 2024-09-16T11:35:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 770206720 bytes 2024-09-16T11:35:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 816957440 bytes 2024-09-16T11:35:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 910476288 bytes 2024-09-16T11:35:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 925888512 bytes 2024-09-16T11:35:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 953049088 bytes 2024-09-16T11:35:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 978519040 bytes 2024-09-16T11:35:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1034843136 bytes 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1071153152 bytes 2024-09-16T11:35:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1128335360 bytes 2024-09-16T11:35:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1171594240 bytes 2024-09-16T11:35:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1225549824 bytes 2024-09-16T11:35:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1246044160 bytes 2024-09-16T11:35:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1297720320 bytes 2024-09-16T11:35:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1357151232 bytes 2024-09-16T11:35:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1449801728 bytes 2024-09-16T11:35:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1514638336 bytes 2024-09-16T11:35:22.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1565913088 bytes 2024-09-16T11:35:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1619028992 bytes 2024-09-16T11:35:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1676561408 bytes 2024-09-16T11:35:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1705844736 bytes 2024-09-16T11:35:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1776808960 bytes 2024-09-16T11:35:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1816467456 bytes 2024-09-16T11:35:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1847708672 bytes 2024-09-16T11:35:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1859774464 bytes 2024-09-16T11:35:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1897791488 bytes 2024-09-16T11:35:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1926985728 bytes 2024-09-16T11:35:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 1953727488 bytes 2024-09-16T11:35:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 1978019840 bytes 2024-09-16T11:35:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 1986694144 bytes 2024-09-16T11:35:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2016569344 bytes 2024-09-16T11:35:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2045684736 bytes 2024-09-16T11:35:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2115279872 bytes 2024-09-16T11:35:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2157464576 bytes 2024-09-16T11:35:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2204249088 bytes 2024-09-16T11:35:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2218183680 bytes 2024-09-16T11:35:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2272728064 bytes 2024-09-16T11:35:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2273357824 bytes 2024-09-16T11:35:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2299368448 bytes 2024-09-16T11:35:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2380010496 bytes 2024-09-16T11:35:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2420389888 bytes 2024-09-16T11:35:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2420994048 bytes 2024-09-16T11:35:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2468264960 bytes 2024-09-16T11:35:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2468899840 bytes 2024-09-16T11:35:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2539389952 bytes 2024-09-16T11:35:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2553812992 bytes 2024-09-16T11:35:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2624279552 bytes 2024-09-16T11:35:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2663863296 bytes 2024-09-16T11:35:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2742707200 bytes 2024-09-16T11:35:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 2743337984 bytes 2024-09-16T11:35:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 2783917056 bytes 2024-09-16T11:35:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 2839605248 bytes 2024-09-16T11:35:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 2902473728 bytes 2024-09-16T11:35:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 2929521664 bytes 2024-09-16T11:35:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 2966825984 bytes 2024-09-16T11:35:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 2997411840 bytes 2024-09-16T11:35:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8200 IOs, 3040090112 bytes 2024-09-16T11:35:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8300 IOs, 3040700416 bytes 2024-09-16T11:35:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8400 IOs, 3078823936 bytes 2024-09-16T11:35:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8500 IOs, 3144276992 bytes 2024-09-16T11:35:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_48 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:35:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:35:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2024-09-16T11:35:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:35:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-16T11:35:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:35:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-16T11:35:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:35:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-16T11:35:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_48': 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 14ae9bc16972 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.14ae9bc16972 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:29 2024 2024-09-16T11:35:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:29 2024 2024-09-16T11:35:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:29 2024 2024-09-16T11:35:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:35:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:35:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 67ab9ab4-c46d-466c-848f-7cc71d56e268 2024-09-16T11:35:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:35:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:35:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-16T11:35:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:35:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:35:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:35:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:35:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:35:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 627712 bytes 2024-09-16T11:35:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 43105280 bytes 2024-09-16T11:35:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 97003520 bytes 2024-09-16T11:35:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 97643520 bytes 2024-09-16T11:35:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 158268416 bytes 2024-09-16T11:35:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 192453632 bytes 2024-09-16T11:35:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 214102016 bytes 2024-09-16T11:35:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 290541568 bytes 2024-09-16T11:35:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 313403392 bytes 2024-09-16T11:35:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 326603776 bytes 2024-09-16T11:35:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 396594176 bytes 2024-09-16T11:35:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 447698944 bytes 2024-09-16T11:35:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 538945536 bytes 2024-09-16T11:35:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 543797248 bytes 2024-09-16T11:35:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 555299840 bytes 2024-09-16T11:35:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 573387776 bytes 2024-09-16T11:35:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 592950272 bytes 2024-09-16T11:35:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 600423424 bytes 2024-09-16T11:35:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 643931136 bytes 2024-09-16T11:35:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 691546112 bytes 2024-09-16T11:35:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 793963520 bytes 2024-09-16T11:35:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 810517504 bytes 2024-09-16T11:35:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 873483264 bytes 2024-09-16T11:35:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 954573824 bytes 2024-09-16T11:35:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 957233152 bytes 2024-09-16T11:35:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 974196736 bytes 2024-09-16T11:35:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1052183552 bytes 2024-09-16T11:35:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1063737344 bytes 2024-09-16T11:35:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1087794176 bytes 2024-09-16T11:35:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1219559424 bytes 2024-09-16T11:35:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1330132992 bytes 2024-09-16T11:35:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1330723840 bytes 2024-09-16T11:35:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1331332096 bytes 2024-09-16T11:35:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1363476480 bytes 2024-09-16T11:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1399975936 bytes 2024-09-16T11:35:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1411151872 bytes 2024-09-16T11:35:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1430956032 bytes 2024-09-16T11:35:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1488898048 bytes 2024-09-16T11:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1538107392 bytes 2024-09-16T11:35:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1569043456 bytes 2024-09-16T11:35:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1677065216 bytes 2024-09-16T11:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1749009408 bytes 2024-09-16T11:35:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1788700672 bytes 2024-09-16T11:35:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1823077376 bytes 2024-09-16T11:35:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1835601920 bytes 2024-09-16T11:35:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1853535232 bytes 2024-09-16T11:35:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1882932224 bytes 2024-09-16T11:35:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1906640896 bytes 2024-09-16T11:35:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1939858432 bytes 2024-09-16T11:35:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1956598784 bytes 2024-09-16T11:35:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1970516992 bytes 2024-09-16T11:35:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1994700800 bytes 2024-09-16T11:35:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2061148160 bytes 2024-09-16T11:35:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2135874560 bytes 2024-09-16T11:35:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2144472064 bytes 2024-09-16T11:35:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2199996416 bytes 2024-09-16T11:35:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2264204288 bytes 2024-09-16T11:35:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2296908800 bytes 2024-09-16T11:35:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2313426944 bytes 2024-09-16T11:35:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2330150912 bytes 2024-09-16T11:35:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2382242816 bytes 2024-09-16T11:35:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2417724416 bytes 2024-09-16T11:35:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2522845184 bytes 2024-09-16T11:35:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2534764544 bytes 2024-09-16T11:35:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2535380992 bytes 2024-09-16T11:35:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2588872704 bytes 2024-09-16T11:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2600673280 bytes 2024-09-16T11:35:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2655063040 bytes 2024-09-16T11:35:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2688433152 bytes 2024-09-16T11:35:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2745521152 bytes 2024-09-16T11:35:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2779148288 bytes 2024-09-16T11:35:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2803934208 bytes 2024-09-16T11:35:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2867272704 bytes 2024-09-16T11:35:42.886 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:35:42.884+0000 7f1e20f72640 -1 reset not still connected to 0x55d52c353110 2024-09-16T11:35:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2939937792 bytes 2024-09-16T11:35:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3001956352 bytes 2024-09-16T11:35:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3046953984 bytes 2024-09-16T11:35:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3084463104 bytes 2024-09-16T11:35:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3127180288 bytes 2024-09-16T11:35:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3165427712 bytes 2024-09-16T11:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3191280640 bytes 2024-09-16T11:35:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3253113856 bytes 2024-09-16T11:35:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_49 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:35:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:35:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:35:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-16T11:35:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:35:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_49': 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 14b7be0e9359 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.14b7be0e9359 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:35:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:44 2024 2024-09-16T11:35:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:44 2024 2024-09-16T11:35:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:44 2024 2024-09-16T11:35:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:35:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:35:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 1040264d-d1a6-45fc-81f5-3534645345d5 2024-09-16T11:35:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:35:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:35:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-16T11:35:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:35:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:35:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=7 2024-09-16T11:35:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:35:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:35:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 19098624 bytes 2024-09-16T11:35:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 71188480 bytes 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 120020992 bytes 2024-09-16T11:35:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 120605696 bytes 2024-09-16T11:35:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 176786432 bytes 2024-09-16T11:35:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 224087040 bytes 2024-09-16T11:35:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 268903424 bytes 2024-09-16T11:35:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 285428736 bytes 2024-09-16T11:35:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 334141440 bytes 2024-09-16T11:35:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 378079232 bytes 2024-09-16T11:35:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 486370304 bytes 2024-09-16T11:35:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 550842368 bytes 2024-09-16T11:35:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 552826880 bytes 2024-09-16T11:35:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 560418816 bytes 2024-09-16T11:35:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 635590656 bytes 2024-09-16T11:35:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 667631616 bytes 2024-09-16T11:35:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 680338432 bytes 2024-09-16T11:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 740276224 bytes 2024-09-16T11:35:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 781594624 bytes 2024-09-16T11:35:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 812794880 bytes 2024-09-16T11:35:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 858692608 bytes 2024-09-16T11:35:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 899803136 bytes 2024-09-16T11:35:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 944715776 bytes 2024-09-16T11:35:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 989452288 bytes 2024-09-16T11:35:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1034884096 bytes 2024-09-16T11:35:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1060068352 bytes 2024-09-16T11:35:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1104312320 bytes 2024-09-16T11:35:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1221894144 bytes 2024-09-16T11:35:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1229282304 bytes 2024-09-16T11:35:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1271200768 bytes 2024-09-16T11:35:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1359728640 bytes 2024-09-16T11:35:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1460036608 bytes 2024-09-16T11:35:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1460627456 bytes 2024-09-16T11:35:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1496568832 bytes 2024-09-16T11:35:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1502383104 bytes 2024-09-16T11:35:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1503332352 bytes 2024-09-16T11:35:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1535001600 bytes 2024-09-16T11:35:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1611261952 bytes 2024-09-16T11:35:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1635299328 bytes 2024-09-16T11:35:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1651767296 bytes 2024-09-16T11:35:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:35:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:35:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:35:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:35:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:35:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_50 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n snapshot ']' 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 128M ']' 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=128M 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:35:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2024-09-16T11:35:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' image = image ']' 2024-09-16T11:35:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-16T11:35:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:35:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:35:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-16T11:35:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:35:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-16T11:35:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_50': 2024-09-16T11:35:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 14c0a525370e 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.14c0a525370e 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:52 2024 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:52 2024 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:52 2024 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:35:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 52447688-b11b-4da0-8dc6-6ae9414561f1 2024-09-16T11:35:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:35:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n '' ']' 2024-09-16T11:35:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-16T11:35:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:35:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:35:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:35:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:35:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=7 2024-09-16T11:35:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:35:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:35:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 10883072 bytes 2024-09-16T11:35:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 58098688 bytes 2024-09-16T11:35:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 137281536 bytes 2024-09-16T11:35:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 188015616 bytes 2024-09-16T11:35:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 204128256 bytes 2024-09-16T11:35:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 247802880 bytes 2024-09-16T11:35:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 281724928 bytes 2024-09-16T11:35:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 335780864 bytes 2024-09-16T11:35:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 336428032 bytes 2024-09-16T11:35:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 404143104 bytes 2024-09-16T11:35:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 532131840 bytes 2024-09-16T11:35:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 577265664 bytes 2024-09-16T11:35:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 638958592 bytes 2024-09-16T11:35:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 753794048 bytes 2024-09-16T11:35:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 790531072 bytes 2024-09-16T11:35:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 831809536 bytes 2024-09-16T11:35:57.084 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T11:35:57.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T11:35:57.085 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T11:35:57.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:35:57.086+0000 7f8142ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 48995) UID: 1000 2024-09-16T11:35:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 929204224 bytes 2024-09-16T11:35:57.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:35:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 1006411776 bytes 2024-09-16T11:35:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1053617152 bytes 2024-09-16T11:35:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1116553216 bytes 2024-09-16T11:35:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1227985920 bytes 2024-09-16T11:35:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1232747520 bytes 2024-09-16T11:35:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1288711168 bytes 2024-09-16T11:35:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1326667776 bytes 2024-09-16T11:35:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1347729408 bytes 2024-09-16T11:35:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1422750720 bytes 2024-09-16T11:35:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1472910336 bytes 2024-09-16T11:35:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1489334272 bytes 2024-09-16T11:35:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1551378432 bytes 2024-09-16T11:35:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1561704448 bytes 2024-09-16T11:35:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1579552768 bytes 2024-09-16T11:35:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1631639552 bytes 2024-09-16T11:35:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1646848000 bytes 2024-09-16T11:36:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1703854080 bytes 2024-09-16T11:36:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1746220032 bytes 2024-09-16T11:36:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1850654720 bytes 2024-09-16T11:36:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1883769856 bytes 2024-09-16T11:36:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_count=50 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local count 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_count=0 2024-09-16T11:36:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 40 2024-09-16T11:36:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:36:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 1 -ne 1 2024-09-16T11:36:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T11:36:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'images: ' 2024-09-16T11:36:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 2 2024-09-16T11:36:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ count=50 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 50 = 50 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state 2024-09-16T11:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 40 2024-09-16T11:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 1 -ne 1 2024-09-16T11:36:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T11:36:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'image health:' 2024-09-16T11:36:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 3 2024-09-16T11:36:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:36:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=WARNING 2024-09-16T11:36:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test WARNING = ERROR 2024-09-16T11:36:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test WARNING = OK 2024-09-16T11:36:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:36:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 2 -ne 1 2024-09-16T11:36:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 30 2024-09-16T11:36:01.086 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T11:36:01.086 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:36:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:36:01.087 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T11:36:01.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:36:01.087 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:36:01.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:36:01.090 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-16T11:36:04.347 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:36:04.343+0000 7f1e20f72640 -1 reset not still connected to 0x55d51ba75d40 2024-09-16T11:36:04.348 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:36:04.343+0000 7f1e20f72640 -1 reset not still connected to 0x55d52a725e10 2024-09-16T11:36:04.348 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:36:04.343+0000 7f1e20f72640 -1 reset not still connected to 0x55d52c64f450 2024-09-16T11:36:04.348 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:36:04.343+0000 7f1e20f72640 -1 reset not still connected to 0x55d5338656c0 2024-09-16T11:36:04.348 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:36:04.343+0000 7f1e20f72640 -1 reset not still connected to 0x55d5347df6c0 2024-09-16T11:36:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T11:36:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'image health:' 2024-09-16T11:36:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 3 2024-09-16T11:36:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:36:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=OK 2024-09-16T11:36:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test OK = ERROR 2024-09-16T11:36:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test OK = OK 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_count=50 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local count 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_count=0 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 40 2024-09-16T11:36:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:36:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 1 -ne 1 2024-09-16T11:36:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T11:36:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'images: ' 2024-09-16T11:36:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 2 2024-09-16T11:36:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:36:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ count=50 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 50 = 50 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state 2024-09-16T11:36:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 40 2024-09-16T11:36:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:36:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 1 -ne 1 2024-09-16T11:36:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T11:36:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'image health:' 2024-09-16T11:36:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 3 2024-09-16T11:36:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=OK 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test OK = ERROR 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test OK = OK 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: compare many images' 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:36:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:36:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:36:31 TEST: compare many images 2024-09-16T11:36:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:36:31 TEST: compare many images 2024-09-16T11:36:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 50 2024-09-16T11:36:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_1 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:36:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-16T11:36:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-16T11:36:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:36:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:36:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:36:33.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:36:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:36:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:36:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:36:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:36:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:36:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-16T11:36:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 4 closest matches: 2024-09-16T11:36:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_14 2024-09-16T11:36:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_16 2024-09-16T11:36:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_18 2024-09-16T11:36:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_19 2024-09-16T11:36:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:36:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-16T11:36:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 3 closest matches: 2024-09-16T11:36:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_10 2024-09-16T11:36:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_15 2024-09-16T11:36:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_17 2024-09-16T11:36:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-16T11:36:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:36:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:36:33.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:36:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:36:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:36:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:36:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:36:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:36:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-16T11:36:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-16T11:36:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:36:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:36:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:36:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:36:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2024-09-16T11:36:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:36:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:36:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:36:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:36:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:36:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:36:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:36:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-16T11:36:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-16T11:36:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_1.status 2024-09-16T11:36:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.status 2024-09-16T11:36:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-16T11:36:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_1.status 2024-09-16T11:36:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-16T11:36:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-16T11:36:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 119 2024-09-16T11:36:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.status 2024-09-16T11:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.status 2024-09-16T11:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-16T11:36:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:36:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=119 2024-09-16T11:36:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:36:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T11:36:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=68 2024-09-16T11:36:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 119 = 68 2024-09-16T11:36:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:36:34.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8618818141314136) > weight (0.25) 2024-09-16T11:36:34.978 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4434916186189949) > weight (0.25) 2024-09-16T11:36:34.978 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.594602266155384) > weight (0.25) 2024-09-16T11:36:34.978 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7040563686195955) > weight (0.25) 2024-09-16T11:36:34.978 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-16T11:36:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T11:36:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=68 2024-09-16T11:36:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 119 = 68 2024-09-16T11:36:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T11:36:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:36:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:36:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:36:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:36:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 119 = '' 2024-09-16T11:36:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.status 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T11:36:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=119 2024-09-16T11:36:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 119 = 119 2024-09-16T11:36:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-16T11:36:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_1': 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c0c604d1b5 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c0c604d1b5 2024-09-16T11:36:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:17 2024 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:17 2024 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:17 2024 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: dca53849-69bc-40ac-a731-5bbd18f66953 2024-09-16T11:36:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.export 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.export 2024-09-16T11:36:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.export 2024-09-16T11:36:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.export 2024-09-16T11:36:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.export 2024-09-16T11:36:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.export 2024-09-16T11:36:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_1.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_1.export 2024-09-16T11:36:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:36:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_2 2024-09-16T11:36:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-16T11:36:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:36:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-16T11:36:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:36:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:36:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:36:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:36:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:36:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-16T11:36:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:36:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:36:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-16T11:36:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 3 closest matches: 2024-09-16T11:36:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_20 2024-09-16T11:36:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_22 2024-09-16T11:36:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_26 2024-09-16T11:36:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:36:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:36:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-16T11:36:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:36:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:36:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:36:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:36:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:36:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_2.status 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.status 2024-09-16T11:36:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-16T11:36:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_2.status 2024-09-16T11:36:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-16T11:36:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-16T11:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 121 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.status 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.status 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-16T11:36:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:36:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=121 2024-09-16T11:36:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T11:36:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=69 2024-09-16T11:36:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 121 = 69 2024-09-16T11:36:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:36:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T11:36:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=69 2024-09-16T11:36:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 121 = 69 2024-09-16T11:36:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:36:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T11:36:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:36:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:36:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:36:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:36:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 121 = '' 2024-09-16T11:36:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:36:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.status 2024-09-16T11:36:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T11:36:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=121 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 121 = 121 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_2': 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11bf4434b277 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11bf4434b277 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:36:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:29 2024 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:29 2024 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:29 2024 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c33e6270-8531-45ef-bdc4-ecc6d15eb898 2024-09-16T11:36:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:36:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.export 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.export 2024-09-16T11:36:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.export 2024-09-16T11:36:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.export 2024-09-16T11:36:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.export 2024-09-16T11:36:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.export 2024-09-16T11:36:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_2.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_2.export 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_3 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:36:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:36:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:36:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:36:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:36:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:36:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:36:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-16T11:36:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 2 closest matches: 2024-09-16T11:36:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_33 2024-09-16T11:36:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_36 2024-09-16T11:36:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-16T11:36:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:36:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:36:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:36:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-16T11:36:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 3 closest matches: 2024-09-16T11:36:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_31 2024-09-16T11:36:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_35 2024-09-16T11:36:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_39 2024-09-16T11:36:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-16T11:36:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:36:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:36:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:36:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:36:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:36:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:36:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:36:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-16T11:36:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 5 closest matches: 2024-09-16T11:36:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_30 2024-09-16T11:36:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_32 2024-09-16T11:36:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_34 2024-09-16T11:36:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_37 2024-09-16T11:36:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_38 2024-09-16T11:36:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:36:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:36:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:36:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:36:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:36:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:36:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:36:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2024-09-16T11:36:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_3.status 2024-09-16T11:36:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.status 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_3.status 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-16T11:36:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 123 2024-09-16T11:36:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.status 2024-09-16T11:36:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.status 2024-09-16T11:36:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-16T11:36:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:36:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=123 2024-09-16T11:36:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T11:36:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=70 2024-09-16T11:36:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 123 = 70 2024-09-16T11:36:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T11:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=70 2024-09-16T11:36:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 123 = 70 2024-09-16T11:36:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T11:36:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:36:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:36:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:36:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:36:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 123 = '' 2024-09-16T11:36:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.status 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=123 2024-09-16T11:36:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 123 = 123 2024-09-16T11:36:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-16T11:36:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_3': 2024-09-16T11:36:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:36:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fcd2917b7f 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fcd2917b7f 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:36 2024 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:36 2024 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:36 2024 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:36:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 4b1d8b43-03c2-4e7d-bdc2-68e83df2c512 2024-09-16T11:36:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.export 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.export 2024-09-16T11:36:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.export 2024-09-16T11:36:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.export 2024-09-16T11:36:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.export 2024-09-16T11:36:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.export 2024-09-16T11:36:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_3.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_3.export 2024-09-16T11:36:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:36:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_4 2024-09-16T11:36:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-16T11:36:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:36:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:36:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-16T11:36:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-16T11:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-16T11:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:36:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:36:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:36:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:36:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:36:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:36:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-16T11:36:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:36:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:36:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:36:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:36:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:36:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:36:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:36:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:36:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-16T11:36:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_4.status 2024-09-16T11:36:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.status 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_4.status 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-16T11:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-16T11:36:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 125 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.status 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.status 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-16T11:36:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:36:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=125 2024-09-16T11:36:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:36:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:36:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:36:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T11:36:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=71 2024-09-16T11:36:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 125 = 71 2024-09-16T11:36:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:36:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:36:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:36:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:36:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:36:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:36:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T11:36:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:36:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:36:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:36:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 125 = '' 2024-09-16T11:36:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:37:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:37:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:37:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T11:37:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 125 = '' 2024-09-16T11:37:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:37:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:37:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:37:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.status 2024-09-16T11:37:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T11:37:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=125 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 125 = 125 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:37:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_4': 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12519c816ea1 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12519c816ea1 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:43 2024 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:43 2024 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:43 2024 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 18f01d1c-b747-4fab-8897-4595d20eb919 2024-09-16T11:37:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-16T11:37:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.export 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.export 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.export 2024-09-16T11:37:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.export 2024-09-16T11:37:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.export 2024-09-16T11:37:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.export 2024-09-16T11:37:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_4.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_4.export 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_5 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-16T11:37:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-16T11:37:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-16T11:37:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-16T11:37:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:37:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:37:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-16T11:37:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:no valid command found; 1 closest matches: 2024-09-16T11:37:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status mirror/image_50 2024-09-16T11:37:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:37:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:37:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:37:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-16T11:37:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-16T11:37:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_5.status 2024-09-16T11:37:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.status 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_5.status 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-16T11:37:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 127 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.status 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.status 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-16T11:37:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=127 2024-09-16T11:37:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T11:37:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=72 2024-09-16T11:37:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 127 = 72 2024-09-16T11:37:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T11:37:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 127 = '' 2024-09-16T11:37:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T11:37:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 127 = '' 2024-09-16T11:37:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.status 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T11:37:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=127 2024-09-16T11:37:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 127 = 127 2024-09-16T11:37:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-16T11:37:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_5': 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c014f2f975 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c014f2f975 2024-09-16T11:37:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:27:56 2024 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:27:56 2024 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:27:56 2024 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 735bbf09-b5b2-4a17-8414-9de64f208e63 2024-09-16T11:37:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.export 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.export 2024-09-16T11:37:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.export 2024-09-16T11:37:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.export 2024-09-16T11:37:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.export 2024-09-16T11:37:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.export 2024-09-16T11:37:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_5.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_5.export 2024-09-16T11:37:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_6 2024-09-16T11:37:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-16T11:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-16T11:37:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-16T11:37:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_6.status 2024-09-16T11:37:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.status 2024-09-16T11:37:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-16T11:37:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_6.status 2024-09-16T11:37:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-16T11:37:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-16T11:37:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 129 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.status 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.status 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-16T11:37:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=129 2024-09-16T11:37:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T11:37:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=73 2024-09-16T11:37:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 129 = 73 2024-09-16T11:37:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T11:37:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 129 = '' 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T11:37:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 129 = '' 2024-09-16T11:37:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:37:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.status 2024-09-16T11:37:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T11:37:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=129 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 129 = 129 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-16T11:37:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_6': 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11bf97a433c 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11bf97a433c 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:03 2024 2024-09-16T11:37:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:03 2024 2024-09-16T11:37:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:03 2024 2024-09-16T11:37:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: bd2d7a6e-4890-4807-895c-c7d327ef0b8d 2024-09-16T11:37:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-16T11:37:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:37:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.export 2024-09-16T11:37:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.export 2024-09-16T11:37:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.export 2024-09-16T11:37:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.export 2024-09-16T11:37:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.export 2024-09-16T11:37:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.export 2024-09-16T11:37:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_6.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_6.export 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_7 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-16T11:37:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-16T11:37:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-16T11:37:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T11:37:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:37:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:37:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-16T11:37:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T11:37:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-09-16T11:37:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:37:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:37:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:37:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:37:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-09-16T11:37:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T11:37:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2024-09-16T11:37:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:37:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:37:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:37:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:37:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:37:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:37:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2024-09-16T11:37:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_7.status 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.status 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_7.status 2024-09-16T11:37:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-16T11:37:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-16T11:37:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 131 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.status 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.status 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-16T11:37:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=131 2024-09-16T11:37:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T11:37:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=74 2024-09-16T11:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 131 = 74 2024-09-16T11:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T11:37:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 131 = '' 2024-09-16T11:37:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T11:37:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 131 = '' 2024-09-16T11:37:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:37:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.status 2024-09-16T11:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T11:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=131 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 131 = 131 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_7': 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc232088e4 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc232088e4 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:14 2024 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:14 2024 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:14 2024 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 215e6e09-36a7-4867-8fa8-8c5aab26f0d4 2024-09-16T11:37:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.export 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.export 2024-09-16T11:37:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.export 2024-09-16T11:37:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.export 2024-09-16T11:37:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.export 2024-09-16T11:37:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.export 2024-09-16T11:37:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_7.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_7.export 2024-09-16T11:37:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_8 2024-09-16T11:37:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-16T11:37:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-16T11:37:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-16T11:37:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-16T11:37:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-16T11:37:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-16T11:37:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_8.status 2024-09-16T11:37:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.status 2024-09-16T11:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-16T11:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_8.status 2024-09-16T11:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-16T11:37:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-16T11:37:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 133 2024-09-16T11:37:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.status 2024-09-16T11:37:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.status 2024-09-16T11:37:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-16T11:37:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=133 2024-09-16T11:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T11:37:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=75 2024-09-16T11:37:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 133 = 75 2024-09-16T11:37:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T11:37:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 133 = '' 2024-09-16T11:37:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.status 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T11:37:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=133 2024-09-16T11:37:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 133 = 133 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-16T11:37:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_8': 2024-09-16T11:37:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 125164f56709 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.125164f56709 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:26 2024 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:26 2024 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:26 2024 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 8f02c93f-0a2c-444a-a327-af8d0dcd2b89 2024-09-16T11:37:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.export 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.export 2024-09-16T11:37:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.export 2024-09-16T11:37:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.export 2024-09-16T11:37:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.export 2024-09-16T11:37:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.export 2024-09-16T11:37:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_8.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_8.export 2024-09-16T11:37:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_9 2024-09-16T11:37:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-16T11:37:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-16T11:37:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-16T11:37:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-16T11:37:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:37:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:37:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-16T11:37:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-16T11:37:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-09-16T11:37:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:37:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:37:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:37:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:37:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-09-16T11:37:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-16T11:37:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-16T11:37:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-16T11:37:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_9.status 2024-09-16T11:37:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.status 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_9.status 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-16T11:37:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 135 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.status 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.status 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-16T11:37:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=135 2024-09-16T11:37:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T11:37:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=76 2024-09-16T11:37:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 135 = 76 2024-09-16T11:37:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T11:37:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 135 = '' 2024-09-16T11:37:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.status 2024-09-16T11:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T11:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=135 2024-09-16T11:37:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 135 = 135 2024-09-16T11:37:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_9': 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c0301dca0a 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c0301dca0a 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:38 2024 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:38 2024 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:38 2024 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: ad8cd2a1-83b6-4d15-ba11-daa8d98ddb16 2024-09-16T11:37:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.export 2024-09-16T11:37:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.export 2024-09-16T11:37:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.export 2024-09-16T11:37:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.export 2024-09-16T11:37:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.export 2024-09-16T11:37:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.export 2024-09-16T11:37:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_9.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_9.export 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_10 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-16T11:37:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-16T11:37:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:37:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:37:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:37:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-16T11:37:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_10.status 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.status 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_10.status 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-16T11:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-16T11:37:45.979 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2802629443513427) > weight (0.25) 2024-09-16T11:37:45.979 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7005379367689754) > weight (0.25) 2024-09-16T11:37:45.979 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8504071483858063) > weight (0.25) 2024-09-16T11:37:45.979 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T11:37:45.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T11:37:45.979 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T11:37:45.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:37:45.978+0000 7fe9c45f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 44123) UID: 1000 2024-09-16T11:37:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 137 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.status 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.status 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-16T11:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=137 2024-09-16T11:37:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:46.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:46.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T11:37:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=77 2024-09-16T11:37:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 137 = 77 2024-09-16T11:37:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T11:37:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 137 = '' 2024-09-16T11:37:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.status 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T11:37:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=137 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 137 = 137 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-16T11:37:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_10': 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11bfe02dc6d6 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11bfe02dc6d6 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:48 2024 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:48 2024 2024-09-16T11:37:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:48 2024 2024-09-16T11:37:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c727cb2e-9ff7-4230-b1cb-e1a3d78fbd0e 2024-09-16T11:37:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-16T11:37:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:37:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.export 2024-09-16T11:37:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.export 2024-09-16T11:37:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.export 2024-09-16T11:37:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.export 2024-09-16T11:37:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.export 2024-09-16T11:37:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.export 2024-09-16T11:37:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_10.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_10.export 2024-09-16T11:37:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_11 2024-09-16T11:37:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-16T11:37:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:49.980 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T11:37:49.980 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:37:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:37:49.981 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T11:37:49.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T11:37:49.981 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T11:37:49.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T11:37:49.985 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-16T11:37:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-16T11:37:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-16T11:37:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-16T11:37:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-16T11:37:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:37:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:37:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-16T11:37:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-16T11:37:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:37:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:37:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:37:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-16T11:37:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-16T11:37:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:37:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:37:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-09-16T11:37:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-16T11:37:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_11.status 2024-09-16T11:37:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.status 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_11.status 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-16T11:37:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 139 2024-09-16T11:37:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.status 2024-09-16T11:37:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.status 2024-09-16T11:37:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-16T11:37:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:37:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=139 2024-09-16T11:37:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:37:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T11:37:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=78 2024-09-16T11:37:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 139 = 78 2024-09-16T11:37:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:37:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T11:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:37:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:37:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:37:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:37:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:37:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 139 = '' 2024-09-16T11:37:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.status 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T11:37:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:37:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:37:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=139 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 139 = 139 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:37:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-16T11:37:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_11': 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc7e107fd5 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc7e107fd5 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:28:57 2024 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:28:57 2024 2024-09-16T11:37:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:28:57 2024 2024-09-16T11:37:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:37:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:37:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:37:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d5fd87f4-333e-453f-8e09-9ee90d26a1c7 2024-09-16T11:37:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:37:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-16T11:37:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:37:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.export 2024-09-16T11:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.export 2024-09-16T11:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.export 2024-09-16T11:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.export 2024-09-16T11:37:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.export 2024-09-16T11:37:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.export 2024-09-16T11:37:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_11.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_11.export 2024-09-16T11:37:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:37:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_12 2024-09-16T11:37:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-16T11:37:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:37:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:37:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:37:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:37:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-16T11:37:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:37:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:37:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:37:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:37:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:37:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:37:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:37:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:37:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:59.094 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6975133453293466) > weight (0.25) 2024-09-16T11:37:59.094 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5760974377965077) > weight (0.25) 2024-09-16T11:37:59.094 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.769611835765252) > weight (0.25) 2024-09-16T11:37:59.094 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8172079460112793) > weight (0.25) 2024-09-16T11:37:59.094 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-16T11:37:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-16T11:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-16T11:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:37:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:37:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-16T11:37:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-16T11:37:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-09-16T11:37:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:37:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:37:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:37:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:37:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:37:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-09-16T11:37:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-16T11:37:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:37:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:37:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:37:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:37:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:37:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:37:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:37:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2024-09-16T11:37:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:37:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:37:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_12.status 2024-09-16T11:37:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.status 2024-09-16T11:37:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-16T11:37:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_12.status 2024-09-16T11:37:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:37:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-16T11:37:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:37:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:37:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:37:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-16T11:38:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 141 2024-09-16T11:38:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.status 2024-09-16T11:38:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:38:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.status 2024-09-16T11:38:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-16T11:38:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=141 2024-09-16T11:38:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:38:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:38:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:38:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T11:38:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=79 2024-09-16T11:38:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 141 = 79 2024-09-16T11:38:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:38:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:38:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:38:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T11:38:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 141 = '' 2024-09-16T11:38:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:38:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:38:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.status 2024-09-16T11:38:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T11:38:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=141 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 141 = 141 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-16T11:38:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_12': 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12514eaefdd6 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12514eaefdd6 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:11 2024 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:11 2024 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:11 2024 2024-09-16T11:38:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: a7a5a597-f597-46c5-bbc6-a5aec2f981b0 2024-09-16T11:38:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.export 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.export 2024-09-16T11:38:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.export 2024-09-16T11:38:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.export 2024-09-16T11:38:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.export 2024-09-16T11:38:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.export 2024-09-16T11:38:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_12.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_12.export 2024-09-16T11:38:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_13 2024-09-16T11:38:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-16T11:38:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-16T11:38:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-16T11:38:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-16T11:38:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-16T11:38:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-16T11:38:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:38:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:38:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:38:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-16T11:38:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-16T11:38:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:38:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-09-16T11:38:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:38:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:38:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:38:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:38:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:38:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:38:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:38:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:38:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:38:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-09-16T11:38:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-16T11:38:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:38:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2024-09-16T11:38:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:38:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:38:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:38:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:38:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:38:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:38:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:38:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2024-09-16T11:38:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:06.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-16T11:38:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_13.status 2024-09-16T11:38:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.status 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_13.status 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-16T11:38:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 143 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.status 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.status 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-16T11:38:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=143 2024-09-16T11:38:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T11:38:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=80 2024-09-16T11:38:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 143 = 80 2024-09-16T11:38:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T11:38:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 143 = '' 2024-09-16T11:38:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.status 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T11:38:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=143 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 143 = 143 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_13': 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0f8eb6e4d 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0f8eb6e4d 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:20 2024 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:20 2024 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:20 2024 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f5a2a94d-7ff1-4c32-9b14-48a803629566 2024-09-16T11:38:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.export 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.export 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.export 2024-09-16T11:38:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.export 2024-09-16T11:38:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.export 2024-09-16T11:38:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.export 2024-09-16T11:38:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_13.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_13.export 2024-09-16T11:38:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_14 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-16T11:38:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-16T11:38:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_14.status 2024-09-16T11:38:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.status 2024-09-16T11:38:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-16T11:38:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_14.status 2024-09-16T11:38:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-16T11:38:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-16T11:38:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 145 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.status 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.status 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-16T11:38:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=145 2024-09-16T11:38:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T11:38:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=81 2024-09-16T11:38:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 145 = 81 2024-09-16T11:38:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T11:38:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 145 = '' 2024-09-16T11:38:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.status 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T11:38:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=145 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 145 = 145 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-16T11:38:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_14': 2024-09-16T11:38:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc6e13ffca 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc6e13ffca 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:36 2024 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:36 2024 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:36 2024 2024-09-16T11:38:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b897dc63-3368-46d2-ae8f-58605bd44840 2024-09-16T11:38:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.export 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.export 2024-09-16T11:38:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.export 2024-09-16T11:38:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.export 2024-09-16T11:38:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.export 2024-09-16T11:38:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.export 2024-09-16T11:38:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_14.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_14.export 2024-09-16T11:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_15 2024-09-16T11:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-16T11:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-16T11:38:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-16T11:38:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-16T11:38:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-16T11:38:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-16T11:38:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:38:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-16T11:38:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:38:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:38:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:38:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:38:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:38:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:38:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:38:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:38:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:38:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:38:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:38:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:38:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:38:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:38:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:38:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:38:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-16T11:38:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-16T11:38:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-16T11:38:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-16T11:38:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_15.status 2024-09-16T11:38:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.status 2024-09-16T11:38:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-16T11:38:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_15.status 2024-09-16T11:38:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-16T11:38:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-16T11:38:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 147 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.status 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.status 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-16T11:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=147 2024-09-16T11:38:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T11:38:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 147 = '' 2024-09-16T11:38:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T11:38:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 147 = '' 2024-09-16T11:38:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.status 2024-09-16T11:38:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T11:38:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=147 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 147 = 147 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-16T11:38:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_15': 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12511deb871e 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12511deb871e 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:47 2024 2024-09-16T11:38:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:47 2024 2024-09-16T11:38:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:47 2024 2024-09-16T11:38:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 8544ced6-19e9-448e-afa3-3117a33e55fb 2024-09-16T11:38:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-16T11:38:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.export 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.export 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.export 2024-09-16T11:38:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.export 2024-09-16T11:38:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.export 2024-09-16T11:38:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.export 2024-09-16T11:38:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_15.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_15.export 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_16 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-16T11:38:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-16T11:38:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-16T11:38:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-16T11:38:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_16.status 2024-09-16T11:38:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.status 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_16.status 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-16T11:38:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 149 2024-09-16T11:38:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.status 2024-09-16T11:38:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.status 2024-09-16T11:38:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-16T11:38:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=149 2024-09-16T11:38:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T11:38:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 149 = '' 2024-09-16T11:38:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T11:38:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 149 = '' 2024-09-16T11:38:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.status 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T11:38:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=149 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 149 = 149 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-16T11:38:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_16': 2024-09-16T11:38:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0479f2966 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0479f2966 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:29:56 2024 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:29:56 2024 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:29:56 2024 2024-09-16T11:38:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b478e757-78fd-4641-8d2d-50753c43edcb 2024-09-16T11:38:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-16T11:38:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:38:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.export 2024-09-16T11:38:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.export 2024-09-16T11:38:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.export 2024-09-16T11:38:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.export 2024-09-16T11:38:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.export 2024-09-16T11:38:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.export 2024-09-16T11:38:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_16.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_16.export 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_17 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-16T11:38:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-16T11:38:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-16T11:38:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:38:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:38:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:38:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:38:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:38:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:38:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:38:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-16T11:38:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-16T11:38:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_17.status 2024-09-16T11:38:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.status 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_17.status 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-16T11:38:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 151 2024-09-16T11:38:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.status 2024-09-16T11:38:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.status 2024-09-16T11:38:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-16T11:38:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=151 2024-09-16T11:38:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T11:38:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 151 = '' 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T11:38:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 151 = '' 2024-09-16T11:38:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.status 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T11:38:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=151 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 151 = 151 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_17': 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c0fda7349f 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c0fda7349f 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:09 2024 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:09 2024 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:09 2024 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: a73d5ffa-b174-4dd8-82e4-e204e403ef6d 2024-09-16T11:38:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-16T11:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.export 2024-09-16T11:38:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.export 2024-09-16T11:38:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.export 2024-09-16T11:38:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.export 2024-09-16T11:38:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.export 2024-09-16T11:38:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.export 2024-09-16T11:38:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_17.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_17.export 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_18 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-16T11:38:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-16T11:38:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-16T11:38:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-16T11:38:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-16T11:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_18.status 2024-09-16T11:38:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.status 2024-09-16T11:38:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-16T11:38:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_18.status 2024-09-16T11:38:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-16T11:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-16T11:38:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 153 2024-09-16T11:38:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.status 2024-09-16T11:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.status 2024-09-16T11:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-16T11:38:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=153 2024-09-16T11:38:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T11:38:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 153 = '' 2024-09-16T11:38:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T11:38:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 153 = '' 2024-09-16T11:38:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.status 2024-09-16T11:38:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T11:38:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=153 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 153 = 153 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-16T11:38:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_18': 2024-09-16T11:38:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fcbd45a8dd 2024-09-16T11:38:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fcbd45a8dd 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:15 2024 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:15 2024 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:15 2024 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b4ce7a6e-9d23-4e33-892a-2b51e7f4088f 2024-09-16T11:38:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-16T11:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.export 2024-09-16T11:38:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.export 2024-09-16T11:38:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.export 2024-09-16T11:38:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.export 2024-09-16T11:38:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.export 2024-09-16T11:38:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.export 2024-09-16T11:38:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_18.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_18.export 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_19 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-16T11:38:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-16T11:38:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-16T11:38:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-16T11:38:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-16T11:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_19.status 2024-09-16T11:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.status 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_19.status 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-16T11:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-16T11:38:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 155 2024-09-16T11:38:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.status 2024-09-16T11:38:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.status 2024-09-16T11:38:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-16T11:38:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=155 2024-09-16T11:38:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T11:38:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=86 2024-09-16T11:38:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 155 = 86 2024-09-16T11:38:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T11:38:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=86 2024-09-16T11:38:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 155 = 86 2024-09-16T11:38:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T11:38:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 155 = '' 2024-09-16T11:38:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:38:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.status 2024-09-16T11:38:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T11:38:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=155 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 155 = 155 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_19': 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1251765e937e 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1251765e937e 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:21 2024 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:21 2024 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:21 2024 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 2d17f5e3-a0dd-42e5-91b8-084a792fce66 2024-09-16T11:38:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.export 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.export 2024-09-16T11:38:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.export 2024-09-16T11:38:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.export 2024-09-16T11:38:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.export 2024-09-16T11:38:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:38:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.export 2024-09-16T11:38:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_19.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_19.export 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_20 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:38:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-16T11:38:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:38:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:38:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:38:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-16T11:38:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:38:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:38:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:38:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:38:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:38:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-16T11:38:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_20.status 2024-09-16T11:38:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.status 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_20.status 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-16T11:38:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-16T11:38:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 157 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.status 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.status 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-16T11:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:38:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=157 2024-09-16T11:38:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T11:38:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 157 = '' 2024-09-16T11:38:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T11:38:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:38:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:38:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:38:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:38:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:38:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 157 = '' 2024-09-16T11:38:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:38:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.status 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T11:38:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:38:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=157 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 157 = 157 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:38:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:38:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_20': 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0b69c2a1e 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0b69c2a1e 2024-09-16T11:38:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:30 2024 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:30 2024 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:30 2024 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 42e976a5-afcc-4873-8194-2f268da32fe5 2024-09-16T11:38:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:38:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.export 2024-09-16T11:38:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.export 2024-09-16T11:38:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.export 2024-09-16T11:38:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.export 2024-09-16T11:39:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.export 2024-09-16T11:39:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.export 2024-09-16T11:39:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_20.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_20.export 2024-09-16T11:39:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_21 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-16T11:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-16T11:39:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:39:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:39:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:39:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-16T11:39:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:39:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:39:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-16T11:39:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-16T11:39:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-16T11:39:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-16T11:39:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-16T11:39:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-09-16T11:39:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-16T11:39:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:39:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2024-09-16T11:39:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_21.status 2024-09-16T11:39:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.status 2024-09-16T11:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-16T11:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_21.status 2024-09-16T11:39:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-16T11:39:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-16T11:39:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 159 2024-09-16T11:39:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.status 2024-09-16T11:39:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.status 2024-09-16T11:39:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-16T11:39:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:39:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=159 2024-09-16T11:39:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T11:39:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 159 = '' 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:39:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T11:39:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 159 = '' 2024-09-16T11:39:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.status 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T11:39:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=159 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 159 = 159 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-16T11:39:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_21': 2024-09-16T11:39:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:39:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:39:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c07ec6e378 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c07ec6e378 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:36 2024 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:36 2024 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:36 2024 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 24a40005-617d-4202-82db-cfe123cbd527 2024-09-16T11:39:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:39:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-16T11:39:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:39:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:39:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.export 2024-09-16T11:39:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.export 2024-09-16T11:39:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.export 2024-09-16T11:39:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.export 2024-09-16T11:39:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.export 2024-09-16T11:39:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.export 2024-09-16T11:39:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_21.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_21.export 2024-09-16T11:39:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_22 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:39:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-16T11:39:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:09.986 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8355066336919864) > weight (0.25) 2024-09-16T11:39:09.986 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.33954935975160394) > weight (0.25) 2024-09-16T11:39:09.986 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7285626659059683) > weight (0.25) 2024-09-16T11:39:09.986 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9122680215743921) > weight (0.25) 2024-09-16T11:39:09.986 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-16T11:39:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-16T11:39:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:39:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-16T11:39:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-16T11:39:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-16T11:39:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:39:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:39:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:39:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:39:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_22.status 2024-09-16T11:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.status 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_22.status 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-16T11:39:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 161 2024-09-16T11:39:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.status 2024-09-16T11:39:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.status 2024-09-16T11:39:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-16T11:39:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:39:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=161 2024-09-16T11:39:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:39:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T11:39:11.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 161 = '' 2024-09-16T11:39:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T11:39:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 161 = '' 2024-09-16T11:39:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:39:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.status 2024-09-16T11:39:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T11:39:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=161 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 161 = 161 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:39:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-16T11:39:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_22': 2024-09-16T11:39:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc4bc93a1f 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc4bc93a1f 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:48 2024 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:48 2024 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:48 2024 2024-09-16T11:39:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:39:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:39:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:39:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: c33407f3-8da2-4b7c-a242-5b063bcddddb 2024-09-16T11:39:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.export 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.export 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.export 2024-09-16T11:39:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.export 2024-09-16T11:39:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.export 2024-09-16T11:39:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.export 2024-09-16T11:39:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_22.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_22.export 2024-09-16T11:39:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_23 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:39:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-16T11:39:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-16T11:39:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:39:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:39:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:39:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:39:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:39:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-16T11:39:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-16T11:39:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-16T11:39:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-16T11:39:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-09-16T11:39:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-09-16T11:39:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:39:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:39:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:39:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:39:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:39:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:39:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:39:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_23.status 2024-09-16T11:39:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.status 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_23.status 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-16T11:39:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 163 2024-09-16T11:39:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.status 2024-09-16T11:39:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.status 2024-09-16T11:39:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-16T11:39:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:39:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=163 2024-09-16T11:39:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:39:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T11:39:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 163 = '' 2024-09-16T11:39:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:39:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T11:39:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 163 = '' 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:39:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.status 2024-09-16T11:39:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T11:39:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=163 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 163 = 163 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:39:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_23': 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1251313033ae 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1251313033ae 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:30:55 2024 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:30:55 2024 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:30:55 2024 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b442bd0b-01b1-474b-ae4e-23213ee88b71 2024-09-16T11:39:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:39:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.export 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.export 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.export 2024-09-16T11:39:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.export 2024-09-16T11:39:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.export 2024-09-16T11:39:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.export 2024-09-16T11:39:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_23.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_23.export 2024-09-16T11:39:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:39:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_24 2024-09-16T11:39:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-16T11:39:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:39:22.096 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T11:39:22.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T11:39:22.096 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T11:39:22.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:39:22.095+0000 7f20287f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 52420) UID: 1000 2024-09-16T11:39:22.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:39:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:39:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-16T11:39:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-16T11:39:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-16T11:39:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:39:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:39:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:39:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:39:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:39:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:39:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:39:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_24.status 2024-09-16T11:39:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.status 2024-09-16T11:39:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-16T11:39:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_24.status 2024-09-16T11:39:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-16T11:39:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-16T11:39:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 165 2024-09-16T11:39:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.status 2024-09-16T11:39:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.status 2024-09-16T11:39:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-16T11:39:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:39:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=165 2024-09-16T11:39:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T11:39:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 165 = '' 2024-09-16T11:39:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:39:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T11:39:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 165 = '' 2024-09-16T11:39:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.status 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T11:39:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=165 2024-09-16T11:39:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 165 = 165 2024-09-16T11:39:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:39:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-16T11:39:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_24': 2024-09-16T11:39:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:39:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:39:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:39:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0180c9988 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0180c9988 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:05 2024 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:05 2024 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:05 2024 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d4a56ee3-400c-40d4-b957-ce8e33caa539 2024-09-16T11:39:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:39:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.export 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.export 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.export 2024-09-16T11:39:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.export 2024-09-16T11:39:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.export 2024-09-16T11:39:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.export 2024-09-16T11:39:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_24.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_24.export 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_25 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:39:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:31.097 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T11:39:31.097 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:39:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:39:31.098 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T11:39:31.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:39:31.098 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:39:31.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:39:31.101 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-16T11:39:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-16T11:39:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:39:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:39:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:39:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:39:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-16T11:39:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-16T11:39:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-16T11:39:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-16T11:39:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-16T11:39:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-16T11:39:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:39:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:39:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-16T11:39:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:39:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:39:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-16T11:39:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-16T11:39:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_25.status 2024-09-16T11:39:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.status 2024-09-16T11:39:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-16T11:39:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_25.status 2024-09-16T11:39:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-16T11:39:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-16T11:39:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 167 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.status 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.status 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-16T11:39:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:39:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=167 2024-09-16T11:39:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T11:39:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=92 2024-09-16T11:39:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 167 = 92 2024-09-16T11:39:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T11:39:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=92 2024-09-16T11:39:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 167 = 92 2024-09-16T11:39:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:39:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T11:39:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 167 = '' 2024-09-16T11:39:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:39:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.status 2024-09-16T11:39:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T11:39:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=167 2024-09-16T11:39:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 167 = 167 2024-09-16T11:39:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:39:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_25': 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aabd2bb3ed 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aabd2bb3ed 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:39:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:20 2024 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:20 2024 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:20 2024 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 72c9a059-c0ba-4ba8-a02d-26f894b5545c 2024-09-16T11:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:39:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.export 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.export 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.export 2024-09-16T11:39:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.export 2024-09-16T11:39:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.export 2024-09-16T11:39:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:39:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.export 2024-09-16T11:39:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_25.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_25.export 2024-09-16T11:39:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:39:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_26 2024-09-16T11:39:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-16T11:39:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:39:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:39:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T11:39:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-16T11:39:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-16T11:39:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:39:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:39:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-16T11:39:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 1 2024-09-16T11:39:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T11:39:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-16T11:39:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-16T11:39:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-16T11:39:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:39:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:39:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:39:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:39:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-16T11:39:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 2 2024-09-16T11:39:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T11:39:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-16T11:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-16T11:39:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-16T11:39:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:39:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:39:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-16T11:39:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 4 2024-09-16T11:39:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T11:39:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T11:39:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:39:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:39:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:39:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-16T11:39:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-16T11:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:39:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:39:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:39:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:39:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-16T11:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:39:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:39:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:39:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:39:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:39:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-16T11:39:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T11:39:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:39:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 8 2024-09-16T11:39:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:39:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:39:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:39:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:39:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:39:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:39:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:39:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:39:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T11:39:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-16T11:39:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:39:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-16T11:39:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_26.status 2024-09-16T11:39:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.status 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_26.status 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-16T11:39:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-16T11:39:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 169 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.status 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.status 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-16T11:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:39:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=169 2024-09-16T11:39:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T11:39:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=93 2024-09-16T11:39:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 169 = 93 2024-09-16T11:39:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:39:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T11:39:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=93 2024-09-16T11:39:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 169 = 93 2024-09-16T11:39:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:39:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T11:39:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:39:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 169 = '' 2024-09-16T11:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:39:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.status 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T11:39:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:39:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=169 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 169 = 169 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:39:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:39:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:39:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:39:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:39:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_26': 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aafff46947 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aafff46947 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:28 2024 2024-09-16T11:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:28 2024 2024-09-16T11:40:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:28 2024 2024-09-16T11:40:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 05dd52ed-bc96-4184-b31a-04d2077723db 2024-09-16T11:40:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.export 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.export 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.export 2024-09-16T11:40:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.export 2024-09-16T11:40:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.export 2024-09-16T11:40:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.export 2024-09-16T11:40:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_26.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_26.export 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_27 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-16T11:40:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-16T11:40:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-16T11:40:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-16T11:40:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:40:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:40:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-16T11:40:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_27.status 2024-09-16T11:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.status 2024-09-16T11:40:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_27.status 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-16T11:40:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 171 2024-09-16T11:40:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.status 2024-09-16T11:40:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.status 2024-09-16T11:40:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-16T11:40:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=171 2024-09-16T11:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T11:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=94 2024-09-16T11:40:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 171 = 94 2024-09-16T11:40:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T11:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=94 2024-09-16T11:40:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 171 = 94 2024-09-16T11:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T11:40:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 171 = '' 2024-09-16T11:40:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.status 2024-09-16T11:40:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T11:40:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=171 2024-09-16T11:40:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 171 = 171 2024-09-16T11:40:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_27': 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0653e8629 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0653e8629 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:41 2024 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:41 2024 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:41 2024 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d7cd8c6a-1904-4edd-946f-03b287d93156 2024-09-16T11:40:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-16T11:40:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:40:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.export 2024-09-16T11:40:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.export 2024-09-16T11:40:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.export 2024-09-16T11:40:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.export 2024-09-16T11:40:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.export 2024-09-16T11:40:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.export 2024-09-16T11:40:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_27.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_27.export 2024-09-16T11:40:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_28 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-16T11:40:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-16T11:40:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-16T11:40:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-16T11:40:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:40:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:40:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:40:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-16T11:40:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_28.status 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.status 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_28.status 2024-09-16T11:40:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-16T11:40:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-16T11:40:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 173 2024-09-16T11:40:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.status 2024-09-16T11:40:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.status 2024-09-16T11:40:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-16T11:40:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=173 2024-09-16T11:40:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T11:40:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:12.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=95 2024-09-16T11:40:12.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 173 = 95 2024-09-16T11:40:12.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T11:40:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=95 2024-09-16T11:40:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 173 = 95 2024-09-16T11:40:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T11:40:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 173 = '' 2024-09-16T11:40:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.status 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T11:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=173 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 173 = 173 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:15.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_28': 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aaefb2bd6a 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aaefb2bd6a 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:31:48 2024 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:31:48 2024 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:31:48 2024 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f168dae1-0384-473d-abfb-2c2c4e9f98dc 2024-09-16T11:40:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.export 2024-09-16T11:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.export 2024-09-16T11:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.export 2024-09-16T11:40:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.export 2024-09-16T11:40:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.export 2024-09-16T11:40:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.export 2024-09-16T11:40:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_28.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_28.export 2024-09-16T11:40:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_29 2024-09-16T11:40:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-16T11:40:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-16T11:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-16T11:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-16T11:40:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-16T11:40:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:40:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:40:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:40:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-16T11:40:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-16T11:40:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:40:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:40:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-09-16T11:40:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-16T11:40:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-09-16T11:40:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:40:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:40:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:40:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:40:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:40:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:40:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:40:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-09-16T11:40:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-16T11:40:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_29.status 2024-09-16T11:40:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.status 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_29.status 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-16T11:40:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 175 2024-09-16T11:40:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.status 2024-09-16T11:40:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.status 2024-09-16T11:40:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-16T11:40:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=175 2024-09-16T11:40:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T11:40:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=96 2024-09-16T11:40:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 175 = 96 2024-09-16T11:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T11:40:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 175 = '' 2024-09-16T11:40:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T11:40:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 175 = '' 2024-09-16T11:40:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.status 2024-09-16T11:40:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T11:40:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=175 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 175 = 175 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_29': 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c02b00f7c0 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c02b00f7c0 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:01 2024 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:01 2024 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:01 2024 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 28f21500-e6e2-438c-ab9f-f105b1869d6b 2024-09-16T11:40:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.export 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.export 2024-09-16T11:40:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.export 2024-09-16T11:40:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.export 2024-09-16T11:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.export 2024-09-16T11:40:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.export 2024-09-16T11:40:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_29.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_29.export 2024-09-16T11:40:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_30 2024-09-16T11:40:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-16T11:40:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-16T11:40:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-16T11:40:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:27.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-16T11:40:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-16T11:40:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-16T11:40:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:40:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:40:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:40:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-16T11:40:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-16T11:40:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:40:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:40:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:40:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-09-16T11:40:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-16T11:40:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-16T11:40:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_30.status 2024-09-16T11:40:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.status 2024-09-16T11:40:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-16T11:40:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_30.status 2024-09-16T11:40:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-16T11:40:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-16T11:40:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 177 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.status 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.status 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-16T11:40:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=177 2024-09-16T11:40:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T11:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=97 2024-09-16T11:40:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 177 = 97 2024-09-16T11:40:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T11:40:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=97 2024-09-16T11:40:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 177 = 97 2024-09-16T11:40:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T11:40:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 177 = '' 2024-09-16T11:40:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.status 2024-09-16T11:40:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T11:40:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=177 2024-09-16T11:40:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 177 = 177 2024-09-16T11:40:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_30': 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc5ba47fc1 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc5ba47fc1 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:14 2024 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:14 2024 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:14 2024 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d6241982-1b6d-4d65-b374-39b9a477b51a 2024-09-16T11:40:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.export 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.export 2024-09-16T11:40:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.export 2024-09-16T11:40:32.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.export 2024-09-16T11:40:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.export 2024-09-16T11:40:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.export 2024-09-16T11:40:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_30.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_30.export 2024-09-16T11:40:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_31 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:33.987 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9449926014554955) > weight (0.25) 2024-09-16T11:40:33.987 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4885563482988301) > weight (0.25) 2024-09-16T11:40:33.987 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T11:40:33.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T11:40:33.988 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T11:40:33.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:40:33.987+0000 7f1f74df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 44820) UID: 1000 2024-09-16T11:40:34.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:40:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-16T11:40:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-16T11:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-16T11:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_31.status 2024-09-16T11:40:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.status 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_31.status 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-16T11:40:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 179 2024-09-16T11:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.status 2024-09-16T11:40:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.status 2024-09-16T11:40:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-16T11:40:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=179 2024-09-16T11:40:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T11:40:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=98 2024-09-16T11:40:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 179 = 98 2024-09-16T11:40:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T11:40:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=98 2024-09-16T11:40:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 179 = 98 2024-09-16T11:40:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T11:40:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 179 = '' 2024-09-16T11:40:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.status 2024-09-16T11:40:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T11:40:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=179 2024-09-16T11:40:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 179 = 179 2024-09-16T11:40:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-16T11:40:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_31': 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a07564b170 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a07564b170 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:25 2024 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:25 2024 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:25 2024 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 57d96513-857d-403d-8a99-bc4b0fd3e473 2024-09-16T11:40:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.export 2024-09-16T11:40:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.export 2024-09-16T11:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.export 2024-09-16T11:40:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.export 2024-09-16T11:40:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.export 2024-09-16T11:40:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.export 2024-09-16T11:40:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_31.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_31.export 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_32 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:41.988 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T11:40:41.988 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:40:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:40:41.989 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T11:40:41.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T11:40:41.990 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T11:40:41.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T11:40:41.995 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-16T11:40:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-16T11:40:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-16T11:40:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-16T11:40:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-16T11:40:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-16T11:40:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-16T11:40:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:40:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:40:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:40:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:40:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-16T11:40:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-16T11:40:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-16T11:40:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:40:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:40:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:40:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:40:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:40:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-16T11:40:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-16T11:40:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_32.status 2024-09-16T11:40:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.status 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_32.status 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-16T11:40:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 181 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.status 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.status 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-16T11:40:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=181 2024-09-16T11:40:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T11:40:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=99 2024-09-16T11:40:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 181 = 99 2024-09-16T11:40:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T11:40:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 181 = '' 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T11:40:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 181 = '' 2024-09-16T11:40:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.status 2024-09-16T11:40:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T11:40:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=181 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 181 = 181 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_32': 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aa94dba686 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aa94dba686 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:35 2024 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:35 2024 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:35 2024 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 905fd472-3e29-4634-b6a3-059626393330 2024-09-16T11:40:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-16T11:40:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.export 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.export 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.export 2024-09-16T11:40:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.export 2024-09-16T11:40:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.export 2024-09-16T11:40:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.export 2024-09-16T11:40:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_32.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_32.export 2024-09-16T11:40:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_33 2024-09-16T11:40:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-16T11:40:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-16T11:40:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-16T11:40:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-16T11:40:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-16T11:40:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:40:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-16T11:40:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:40:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:40:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:40:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:40:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-16T11:40:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-16T11:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_33.status 2024-09-16T11:40:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.status 2024-09-16T11:40:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-16T11:40:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_33.status 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-16T11:40:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 183 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.status 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.status 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-16T11:40:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:40:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=183 2024-09-16T11:40:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:40:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T11:40:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=100 2024-09-16T11:40:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 183 = 100 2024-09-16T11:40:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:40:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T11:40:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 183 = '' 2024-09-16T11:40:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:40:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T11:40:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:40:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 183 = '' 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.status 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T11:40:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:40:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=183 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 183 = 183 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-16T11:40:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_33': 2024-09-16T11:40:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:40:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:40:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:40:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c0828940e4 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c0828940e4 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:42 2024 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:42 2024 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:42 2024 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d843e793-ea22-4d9f-a27b-0c60b2326370 2024-09-16T11:40:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.export 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.export 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.export 2024-09-16T11:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.export 2024-09-16T11:40:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.export 2024-09-16T11:40:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:40:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.export 2024-09-16T11:40:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_33.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_33.export 2024-09-16T11:40:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:40:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_34 2024-09-16T11:40:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-16T11:40:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:40:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:40:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:40:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:40:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-16T11:40:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:40:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:40:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:40:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:40:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:40:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:40:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:40:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-16T11:40:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:40:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:40:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:40:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:40:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:40:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:40:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:40:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:40:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_34.status 2024-09-16T11:40:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.status 2024-09-16T11:40:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-16T11:40:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_34.status 2024-09-16T11:40:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:40:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:40:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-16T11:40:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:40:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:40:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:40:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-16T11:41:00.102 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.46338873751245324) > weight (0.25) 2024-09-16T11:41:00.102 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3408589899169192) > weight (0.25) 2024-09-16T11:41:00.102 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9171892692716502) > weight (0.25) 2024-09-16T11:41:00.102 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-16T11:41:00.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-16T11:41:00.103 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T11:41:00.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:41:00.102+0000 7fc3fd3f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 51604) UID: 1000 2024-09-16T11:41:00.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:41:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 185 2024-09-16T11:41:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.status 2024-09-16T11:41:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.status 2024-09-16T11:41:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-16T11:41:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:41:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=185 2024-09-16T11:41:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T11:41:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=101 2024-09-16T11:41:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 185 = 101 2024-09-16T11:41:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:41:01.103 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-16T11:41:01.104 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:41:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:41:01.104 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-16T11:41:01.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T11:41:01.104 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T11:41:01.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T11:41:01.107 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-16T11:41:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T11:41:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 185 = '' 2024-09-16T11:41:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T11:41:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 185 = '' 2024-09-16T11:41:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.status 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T11:41:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=185 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 185 = 185 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_34': 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc36f2581b 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc36f2581b 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:41:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:48 2024 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:48 2024 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:48 2024 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 6abecb8f-c955-47d0-8423-8d9b6c3ccf30 2024-09-16T11:41:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.export 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.export 2024-09-16T11:41:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.export 2024-09-16T11:41:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.export 2024-09-16T11:41:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.export 2024-09-16T11:41:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.export 2024-09-16T11:41:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_34.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_34.export 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_35 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:41:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:41:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:41:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:41:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:41:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-16T11:41:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:41:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:41:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-16T11:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_35.status 2024-09-16T11:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.status 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_35.status 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-16T11:41:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 187 2024-09-16T11:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.status 2024-09-16T11:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.status 2024-09-16T11:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-16T11:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:41:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=187 2024-09-16T11:41:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T11:41:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=102 2024-09-16T11:41:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 187 = 102 2024-09-16T11:41:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:41:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T11:41:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 187 = '' 2024-09-16T11:41:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.status 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T11:41:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=187 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 187 = 187 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_35': 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0ac68b65 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0ac68b65 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:41:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:32:59 2024 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:32:59 2024 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:32:59 2024 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 22d639e5-b167-4825-bd14-2050baefdab4 2024-09-16T11:41:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:41:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.export 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.export 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.export 2024-09-16T11:41:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.export 2024-09-16T11:41:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.export 2024-09-16T11:41:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.export 2024-09-16T11:41:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_35.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_35.export 2024-09-16T11:41:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_36 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:41:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-16T11:41:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-16T11:41:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:41:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:41:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:41:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-16T11:41:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:41:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:41:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:41:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:41:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:41:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T11:41:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T11:41:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T11:41:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:41:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:41:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-16T11:41:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 1 2024-09-16T11:41:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T11:41:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T11:41:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T11:41:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:41:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:41:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-16T11:41:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 2 2024-09-16T11:41:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T11:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T11:41:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T11:41:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T11:41:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:41:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:41:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:41:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:41:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-16T11:41:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 4 2024-09-16T11:41:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T11:41:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T11:41:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T11:41:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T11:41:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T11:41:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:41:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:41:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:41:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:41:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:41:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:41:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-16T11:41:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 8 2024-09-16T11:41:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T11:41:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T11:41:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T11:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T11:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T11:41:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:41:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:41:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:41:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:41:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:41:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-16T11:41:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_36.status 2024-09-16T11:41:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.status 2024-09-16T11:41:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-16T11:41:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_36.status 2024-09-16T11:41:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-16T11:41:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-16T11:41:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 189 2024-09-16T11:41:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.status 2024-09-16T11:41:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.status 2024-09-16T11:41:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-16T11:41:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=189 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:41:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T11:41:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=103 2024-09-16T11:41:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 189 = 103 2024-09-16T11:41:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:41:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T11:41:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 189 = '' 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.status 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T11:41:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=189 2024-09-16T11:41:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 189 = 189 2024-09-16T11:41:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-16T11:41:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_36': 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aa2ef04aa6 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aa2ef04aa6 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:08 2024 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:08 2024 2024-09-16T11:41:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:08 2024 2024-09-16T11:41:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:41:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:41:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:41:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 5caa0b20-60a7-472a-a287-3766f979eba6 2024-09-16T11:41:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:41:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-16T11:41:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:41:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:41:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.export 2024-09-16T11:41:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.export 2024-09-16T11:41:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.export 2024-09-16T11:41:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.export 2024-09-16T11:41:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.export 2024-09-16T11:41:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.export 2024-09-16T11:41:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_36.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_36.export 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_37 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:41:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:41:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-16T11:41:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:41:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:41:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:41:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-16T11:41:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-16T11:41:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-16T11:41:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-16T11:41:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-16T11:41:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-16T11:41:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-16T11:41:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:41:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:41:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:41:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-16T11:41:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_37.status 2024-09-16T11:41:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.status 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_37.status 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-16T11:41:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 191 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.status 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.status 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-16T11:41:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:41:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=191 2024-09-16T11:41:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T11:41:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=104 2024-09-16T11:41:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 191 = 104 2024-09-16T11:41:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:41:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T11:41:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 191 = '' 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:41:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.status 2024-09-16T11:41:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T11:41:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=191 2024-09-16T11:41:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 191 = 191 2024-09-16T11:41:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-16T11:41:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_37': 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c040654ee1 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c040654ee1 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:23 2024 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:23 2024 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:23 2024 2024-09-16T11:41:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:41:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:41:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:41:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: eb944d6a-f8cb-4ae2-a094-e3ad1747c24a 2024-09-16T11:41:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:41:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.export 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.export 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.export 2024-09-16T11:41:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.export 2024-09-16T11:41:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.export 2024-09-16T11:41:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.export 2024-09-16T11:41:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_37.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_37.export 2024-09-16T11:41:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_38 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-16T11:41:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:41:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-16T11:41:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:41:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:41:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:41:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_38.status 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.status 2024-09-16T11:41:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_38.status 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-16T11:41:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 193 2024-09-16T11:41:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.status 2024-09-16T11:41:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.status 2024-09-16T11:41:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-16T11:41:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:41:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=193 2024-09-16T11:41:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:41:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T11:41:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=105 2024-09-16T11:41:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 193 = 105 2024-09-16T11:41:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:41:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T11:41:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 193 = '' 2024-09-16T11:41:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:41:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.status 2024-09-16T11:41:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T11:41:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=193 2024-09-16T11:41:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 193 = 193 2024-09-16T11:41:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-16T11:41:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-16T11:41:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_38': 2024-09-16T11:41:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fcdf75b57f 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fcdf75b57f 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:36 2024 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:36 2024 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:36 2024 2024-09-16T11:41:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:41:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:41:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:41:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 7445f68a-3ed3-43b0-b798-83462bd9fca7 2024-09-16T11:41:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:41:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-16T11:41:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:41:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:41:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.export 2024-09-16T11:41:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.export 2024-09-16T11:41:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.export 2024-09-16T11:41:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.export 2024-09-16T11:41:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.export 2024-09-16T11:41:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.export 2024-09-16T11:41:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_38.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_38.export 2024-09-16T11:41:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_39 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:41:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-16T11:41:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:41:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:41:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-16T11:41:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-16T11:41:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-16T11:41:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-16T11:41:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-16T11:41:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-16T11:41:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:41:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:41:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-16T11:41:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-16T11:41:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_39.status 2024-09-16T11:41:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.status 2024-09-16T11:41:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-16T11:41:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_39.status 2024-09-16T11:41:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-16T11:41:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-16T11:41:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 195 2024-09-16T11:41:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.status 2024-09-16T11:41:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.status 2024-09-16T11:41:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-16T11:41:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:41:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=195 2024-09-16T11:41:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T11:41:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=106 2024-09-16T11:41:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 195 = 106 2024-09-16T11:41:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:41:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T11:41:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:41:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:41:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:41:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:41:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:41:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 195 = '' 2024-09-16T11:41:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:41:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.status 2024-09-16T11:41:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T11:41:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:41:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=195 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 195 = 195 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-16T11:41:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_39': 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a058a37b0f 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a058a37b0f 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:43 2024 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:43 2024 2024-09-16T11:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:43 2024 2024-09-16T11:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: e54108d2-02f3-4f44-b0c9-f084887582f4 2024-09-16T11:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-16T11:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.export 2024-09-16T11:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.export 2024-09-16T11:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.export 2024-09-16T11:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.export 2024-09-16T11:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.export 2024-09-16T11:41:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.export 2024-09-16T11:41:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_39.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_39.export 2024-09-16T11:41:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:41:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_40 2024-09-16T11:41:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-16T11:41:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:41:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:41:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-16T11:41:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-16T11:41:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:41:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:41:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:41:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-16T11:41:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:41:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:41:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:41:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:41:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:41:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:41:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-16T11:41:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:41:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-16T11:41:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-16T11:41:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-16T11:41:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:41:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:41:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-16T11:41:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-16T11:41:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:41:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-16T11:41:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:41:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:41:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:41:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:41:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:41:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:41:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-16T11:41:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:41:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:41:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_40.status 2024-09-16T11:41:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.status 2024-09-16T11:41:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_40.status 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:41:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-16T11:42:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 197 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.status 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.status 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:42:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=197 2024-09-16T11:42:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:42:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:42:01.107 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6215104571639402) > weight (0.25) 2024-09-16T11:42:01.108 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8415242703935408) > weight (0.25) 2024-09-16T11:42:01.108 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T11:42:01.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T11:42:01.108 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T11:42:01.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:42:01.107+0000 7f5641ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 44129) UID: 1000 2024-09-16T11:42:01.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:42:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:42:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 2 2024-09-16T11:42:03.995 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2883119063922871) > weight (0.25) 2024-09-16T11:42:03.995 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6767008762991027) > weight (0.25) 2024-09-16T11:42:03.996 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.44553182792812995) > weight (0.25) 2024-09-16T11:42:03.996 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8107419954662534) > weight (0.25) 2024-09-16T11:42:03.996 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 2 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 4 2024-09-16T11:42:10.109 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T11:42:10.109 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:42:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:42:10.110 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T11:42:10.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T11:42:10.110 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T11:42:10.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T11:42:10.113 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 4 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 8 2024-09-16T11:42:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 8 2024-09-16T11:42:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = '' 2024-09-16T11:42:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 16 2024-09-16T11:42:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.status 2024-09-16T11:42:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T11:42:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=197 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 197 = 197 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:42:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:42:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-16T11:42:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_40': 2024-09-16T11:42:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aa6a45e6a5 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aa6a45e6a5 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:33:56 2024 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:33:56 2024 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:33:56 2024 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:42:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:42:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:42:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: de17e228-9a77-43f1-bfec-27661cda333e 2024-09-16T11:42:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.export 2024-09-16T11:42:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.export 2024-09-16T11:42:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.export 2024-09-16T11:42:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.export 2024-09-16T11:42:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.export 2024-09-16T11:42:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.export 2024-09-16T11:42:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_40.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_40.export 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_41 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:42:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:42:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:42:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:42:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:42:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-16T11:42:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:42:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:42:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-16T11:42:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T11:42:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:42:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-16T11:42:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:42:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:42:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:42:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-16T11:42:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T11:42:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:42:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-16T11:42:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:42:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:42:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:42:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:42:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:42:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-16T11:42:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T11:42:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:42:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-16T11:42:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:42:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:42:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:42:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:42:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:42:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:42:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-16T11:42:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:42:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:42:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:42:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_41.status 2024-09-16T11:42:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.status 2024-09-16T11:42:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-16T11:42:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_41.status 2024-09-16T11:42:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-16T11:42:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-16T11:42:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 199 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.status 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.status 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-16T11:42:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:42:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=199 2024-09-16T11:42:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T11:42:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=108 2024-09-16T11:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 199 = 108 2024-09-16T11:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:42:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T11:42:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=108 2024-09-16T11:42:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 199 = 108 2024-09-16T11:42:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T11:42:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:42:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:42:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:42:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:42:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:42:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 199 = '' 2024-09-16T11:42:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:42:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.status 2024-09-16T11:42:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T11:42:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:42:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=199 2024-09-16T11:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 199 = 199 2024-09-16T11:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-16T11:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-16T11:42:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_41': 2024-09-16T11:42:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:42:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:42:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:42:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 10c02f0100c5 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.10c02f0100c5 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:06 2024 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:06 2024 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:06 2024 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 316bb7bf-0be4-4d21-b8d9-199ffb446933 2024-09-16T11:42:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:42:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-16T11:42:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:42:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:42:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:42:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.export 2024-09-16T11:42:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.export 2024-09-16T11:42:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.export 2024-09-16T11:42:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.export 2024-09-16T11:42:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.export 2024-09-16T11:42:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.export 2024-09-16T11:42:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_41.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_41.export 2024-09-16T11:42:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:42:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_42 2024-09-16T11:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-16T11:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:42:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:42:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:42:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-16T11:42:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-16T11:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-16T11:43:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-16T11:43:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_42.status 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.status 2024-09-16T11:43:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-16T11:43:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_42.status 2024-09-16T11:43:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-16T11:43:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-16T11:43:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 201 2024-09-16T11:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.status 2024-09-16T11:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.status 2024-09-16T11:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-16T11:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=201 2024-09-16T11:43:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T11:43:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=109 2024-09-16T11:43:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 201 = 109 2024-09-16T11:43:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T11:43:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 201 = '' 2024-09-16T11:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T11:43:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 201 = '' 2024-09-16T11:43:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.status 2024-09-16T11:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T11:43:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=201 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 201 = 201 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_42': 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc96ac4332 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc96ac4332 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:19 2024 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:19 2024 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:19 2024 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f10dfeca-d8b5-4422-b71a-0f86c6765b14 2024-09-16T11:43:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.export 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.export 2024-09-16T11:43:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.export 2024-09-16T11:43:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.export 2024-09-16T11:43:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.export 2024-09-16T11:43:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.export 2024-09-16T11:43:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_42.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_42.export 2024-09-16T11:43:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_43 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-16T11:43:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-16T11:43:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-16T11:43:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-16T11:43:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-16T11:43:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:43:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:43:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-16T11:43:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-16T11:43:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:43:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:43:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:43:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2024-09-16T11:43:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-16T11:43:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_43.status 2024-09-16T11:43:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.status 2024-09-16T11:43:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-16T11:43:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_43.status 2024-09-16T11:43:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-16T11:43:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-16T11:43:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 203 2024-09-16T11:43:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.status 2024-09-16T11:43:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.status 2024-09-16T11:43:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-16T11:43:10.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=203 2024-09-16T11:43:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T11:43:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=110 2024-09-16T11:43:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 203 = 110 2024-09-16T11:43:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T11:43:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=110 2024-09-16T11:43:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 203 = 110 2024-09-16T11:43:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T11:43:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 203 = '' 2024-09-16T11:43:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:12.548 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:43:12.546+0000 7f48ae988640 -1 reset not still connected to 0x5559797ee8f0 2024-09-16T11:43:12.548 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:43:12.546+0000 7f48ae988640 -1 reset not still connected to 0x5559797eeea0 2024-09-16T11:43:12.549 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:43:12.546+0000 7f48ae988640 -1 reset not still connected to 0x55597aabd790 2024-09-16T11:43:12.549 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:43:12.546+0000 7f48ae988640 -1 reset not still connected to 0x55597d219930 2024-09-16T11:43:12.549 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:43:12.546+0000 7f48ae988640 -1 reset not still connected to 0x55599b7e2680 2024-09-16T11:43:12.549 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:43:12.546+0000 7f48ae988640 -1 reset not still connected to 0x5559b45b5c70 2024-09-16T11:43:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.status 2024-09-16T11:43:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T11:43:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=203 2024-09-16T11:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 203 = 203 2024-09-16T11:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-16T11:43:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-16T11:43:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_43': 2024-09-16T11:43:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12ba395f902b 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12ba395f902b 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:33 2024 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:33 2024 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:33 2024 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 6eadc152-f19b-4a5d-abbd-4c03a784ef9f 2024-09-16T11:43:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.export 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.export 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.export 2024-09-16T11:43:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.export 2024-09-16T11:43:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.export 2024-09-16T11:43:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.export 2024-09-16T11:43:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_43.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_43.export 2024-09-16T11:43:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_44 2024-09-16T11:43:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-16T11:43:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-16T11:43:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-16T11:43:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-16T11:43:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-16T11:43:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-16T11:43:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-16T11:43:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-16T11:43:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:43:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:43:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:43:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:43:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-16T11:43:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-16T11:43:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:43:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:43:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:43:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2024-09-16T11:43:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_44.status 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.status 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_44.status 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-16T11:43:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 205 2024-09-16T11:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.status 2024-09-16T11:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.status 2024-09-16T11:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-16T11:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=205 2024-09-16T11:43:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T11:43:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=111 2024-09-16T11:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 205 = 111 2024-09-16T11:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T11:43:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=111 2024-09-16T11:43:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 205 = 111 2024-09-16T11:43:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T11:43:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 205 = '' 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.status 2024-09-16T11:43:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T11:43:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=205 2024-09-16T11:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 205 = 205 2024-09-16T11:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-16T11:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-16T11:43:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_44': 2024-09-16T11:43:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12ba8a2dda37 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12ba8a2dda37 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:43 2024 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:43 2024 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:43 2024 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 0aab3740-d202-4c69-8377-5da1f823faf0 2024-09-16T11:43:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.export 2024-09-16T11:43:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.export 2024-09-16T11:43:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.export 2024-09-16T11:43:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.export 2024-09-16T11:43:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.export 2024-09-16T11:43:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.export 2024-09-16T11:43:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_44.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_44.export 2024-09-16T11:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_45 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-16T11:43:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-16T11:43:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-16T11:43:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-16T11:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-16T11:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:43:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:43:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:43:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:43:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-16T11:43:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-16T11:43:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-09-16T11:43:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:43:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:43:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:43:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:43:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:43:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:43:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:43:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:43:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-09-16T11:43:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-16T11:43:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-16T11:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_45.status 2024-09-16T11:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.status 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_45.status 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-16T11:43:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 207 2024-09-16T11:43:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.status 2024-09-16T11:43:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.status 2024-09-16T11:43:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-16T11:43:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=207 2024-09-16T11:43:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T11:43:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=112 2024-09-16T11:43:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 207 = 112 2024-09-16T11:43:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T11:43:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 207 = '' 2024-09-16T11:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T11:43:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 207 = '' 2024-09-16T11:43:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.status 2024-09-16T11:43:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T11:43:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=207 2024-09-16T11:43:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 207 = 207 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-16T11:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_45': 2024-09-16T11:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12baadbf7308 2024-09-16T11:43:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12baadbf7308 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:34:56 2024 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:34:56 2024 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:34:56 2024 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 4de1128c-eeed-43f2-8404-66dfce23f601 2024-09-16T11:43:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.export 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.export 2024-09-16T11:43:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.export 2024-09-16T11:43:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.export 2024-09-16T11:43:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.export 2024-09-16T11:43:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.export 2024-09-16T11:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_45.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_45.export 2024-09-16T11:43:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_46 2024-09-16T11:43:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-16T11:43:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-16T11:43:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-16T11:43:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-16T11:43:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-16T11:43:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-16T11:43:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T11:43:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-16T11:43:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T11:43:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-16T11:43:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:43:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:43:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:43:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-16T11:43:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T11:43:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-09-16T11:43:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:43:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:43:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:43:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:43:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:43:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:43:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-09-16T11:43:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-16T11:43:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-16T11:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_46.status 2024-09-16T11:43:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.status 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_46.status 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-16T11:43:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-16T11:43:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 209 2024-09-16T11:43:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.status 2024-09-16T11:43:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.status 2024-09-16T11:43:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-16T11:43:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=209 2024-09-16T11:43:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T11:43:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=113 2024-09-16T11:43:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 209 = 113 2024-09-16T11:43:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T11:43:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 209 = '' 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:35.997 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T11:43:35.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T11:43:35.998 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T11:43:35.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:43:35.997+0000 7f6034ff9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 50895) UID: 1000 2024-09-16T11:43:36.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T11:43:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 209 = '' 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.status 2024-09-16T11:43:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T11:43:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=209 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 209 = 209 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-16T11:43:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_46': 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a032a5d057 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a032a5d057 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:06 2024 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:06 2024 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:06 2024 2024-09-16T11:43:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 541cb1e3-d977-4c95-ba41-c949127ff0cd 2024-09-16T11:43:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-16T11:43:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:43:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.export 2024-09-16T11:43:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.export 2024-09-16T11:43:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.export 2024-09-16T11:43:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.export 2024-09-16T11:43:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.export 2024-09-16T11:43:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.export 2024-09-16T11:43:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_46.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_46.export 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_47 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:41.115 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T11:43:41.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T11:43:41.116 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T11:43:41.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:43:41.115+0000 7fa9f3df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 58255) UID: 1000 2024-09-16T11:43:41.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:43:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-16T11:43:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-16T11:43:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-16T11:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-16T11:43:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:43:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:43:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:43:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-16T11:43:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-16T11:43:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_47.status 2024-09-16T11:43:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.status 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_47.status 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-16T11:43:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 211 2024-09-16T11:43:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.status 2024-09-16T11:43:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.status 2024-09-16T11:43:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-16T11:43:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=211 2024-09-16T11:43:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T11:43:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=114 2024-09-16T11:43:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 211 = 114 2024-09-16T11:43:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T11:43:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 211 = '' 2024-09-16T11:43:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:44.998 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T11:43:44.999 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:43:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:43:44.999 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T11:43:44.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T11:43:44.999 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T11:43:45.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T11:43:45.002 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T11:43:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 211 = '' 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.status 2024-09-16T11:43:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T11:43:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=211 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 211 = 211 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-16T11:43:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_47': 2024-09-16T11:43:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12aae0c1c94a 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12aae0c1c94a 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:17 2024 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:17 2024 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:17 2024 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: a52e8873-193f-487c-b67c-f482a83709db 2024-09-16T11:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.export 2024-09-16T11:43:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.export 2024-09-16T11:43:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.export 2024-09-16T11:43:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.export 2024-09-16T11:43:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.export 2024-09-16T11:43:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.export 2024-09-16T11:43:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_47.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_47.export 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_48 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:49.116 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T11:43:49.116 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:43:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:43:49.117 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T11:43:49.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:43:49.117 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:43:49.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:43:49.120 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-16T11:43:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-16T11:43:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:50.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-16T11:43:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_48.status 2024-09-16T11:43:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.status 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_48.status 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-16T11:43:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 213 2024-09-16T11:43:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.status 2024-09-16T11:43:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.status 2024-09-16T11:43:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-16T11:43:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=213 2024-09-16T11:43:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T11:43:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=115 2024-09-16T11:43:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 213 = 115 2024-09-16T11:43:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T11:43:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=115 2024-09-16T11:43:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 213 = 115 2024-09-16T11:43:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:43:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T11:43:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:43:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:43:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:43:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:43:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 213 = '' 2024-09-16T11:43:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:43:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.status 2024-09-16T11:43:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T11:43:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:43:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=213 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 213 = 213 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:43:54.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-16T11:43:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_48': 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12bafd6df039 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12bafd6df039 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:32 2024 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:32 2024 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:32 2024 2024-09-16T11:43:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:43:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:43:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:43:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 67ab9ab4-c46d-466c-848f-7cc71d56e268 2024-09-16T11:43:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:43:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.export 2024-09-16T11:43:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.export 2024-09-16T11:43:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.export 2024-09-16T11:43:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.export 2024-09-16T11:43:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.export 2024-09-16T11:43:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.export 2024-09-16T11:43:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_48.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_48.export 2024-09-16T11:43:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_49 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:43:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:43:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:43:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:43:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-16T11:43:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:43:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-16T11:43:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-16T11:43:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:43:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-16T11:43:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-16T11:43:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:43:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:43:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:43:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-16T11:43:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-16T11:43:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:43:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2024-09-16T11:43:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:43:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:43:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:43:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:43:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:43:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2024-09-16T11:43:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:43:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:43:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:43:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_49.status 2024-09-16T11:43:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.status 2024-09-16T11:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-16T11:43:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_49.status 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-16T11:43:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 215 2024-09-16T11:43:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.status 2024-09-16T11:43:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.status 2024-09-16T11:43:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-16T11:43:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:43:59.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:43:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=215 2024-09-16T11:43:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:43:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T11:43:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=116 2024-09-16T11:44:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 215 = 116 2024-09-16T11:44:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T11:44:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=116 2024-09-16T11:44:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 215 = 116 2024-09-16T11:44:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T11:44:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:44:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:44:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:44:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:44:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:44:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 215 = '' 2024-09-16T11:44:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:44:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:44:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:44:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.status 2024-09-16T11:44:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T11:44:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=215 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 215 = 215 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:44:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:44:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:44:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:44:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:44:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_49': 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 11fc55ed93b4 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.11fc55ed93b4 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:44:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:47 2024 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:47 2024 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:47 2024 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 1040264d-d1a6-45fc-81f5-3534645345d5 2024-09-16T11:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:44:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-16T11:44:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:44:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:44:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:44:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.export 2024-09-16T11:44:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.export 2024-09-16T11:44:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.export 2024-09-16T11:44:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.export 2024-09-16T11:44:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:44:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.export 2024-09-16T11:44:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:44:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.export 2024-09-16T11:44:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_49.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_49.export 2024-09-16T11:44:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_50 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:44:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T11:44:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-16T11:44:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=started 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:44:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=started 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_result 2024-09-16T11:44:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=stopped 2024-09-16T11:44:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-16T11:44:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_instance=cluster1 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -i state 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance=cluster1 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local loop_instance 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sleep 0 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:44:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ seq 0 3 2024-09-16T11:44:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:0 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:0 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:44:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=0 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:0 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=0 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:44:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:44:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:44:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=0 2024-09-16T11:44:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=0 2024-09-16T11:44:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:44:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-16T11:44:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-16T11:44:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:44:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:44:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:1 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:1 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=1 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:44:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=1 2024-09-16T11:44:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:44:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:44:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:44:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=1 2024-09-16T11:44:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=1 2024-09-16T11:44:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:44:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-16T11:44:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-16T11:44:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:44:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:2 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T11:44:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:2 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=2 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=2 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:2 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T11:44:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:2 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=2 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=2 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=2 2024-09-16T11:44:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T11:44:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:44:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:44:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T11:44:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-16T11:44:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-16T11:44:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:44:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:admin_socket: invalid command 2024-09-16T11:44:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1:3 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ shift 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local val=cluster1:3 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local instance_var_name=instance 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=3 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T11:44:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval cluster=cluster1 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ eval instance=3 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local local_cluster=cluster1:3 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster=cluster1 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local val=cluster1:3 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ cluster=cluster1 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ instance=3 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ local_cluster=cluster1 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ eval instance=3 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ instance=3 2024-09-16T11:44:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T11:44:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T11:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-09-16T11:44:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ return 0 2024-09-16T11:44:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:44:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:44:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ grep -i Replaying 2024-09-16T11:44:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: "state": "Replaying" 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=started 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test started = started 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = journal ']' 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_cluster=cluster1 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-16T11:44:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster2-mirror-image_50.status 2024-09-16T11:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local status_log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.status 2024-09-16T11:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-16T11:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ echo cluster1-mirror-image_50.status 2024-09-16T11:44:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:44:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local local_status_log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-16T11:44:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-16T11:44:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Snapshot ID: 217 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.status 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.status 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-16T11:44:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T11:44:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snapshot_id=217 2024-09-16T11:44:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:44:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.2 2024-09-16T11:44:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T11:44:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=117 2024-09-16T11:44:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 217 = 117 2024-09-16T11:44:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.4 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T11:44:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=117 2024-09-16T11:44:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 217 = 117 2024-09-16T11:44:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.8 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T11:44:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:44:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:-:1.1: Document is empty 2024-09-16T11:44:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: 2024-09-16T11:44:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:^ 2024-09-16T11:44:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id= 2024-09-16T11:44:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 217 = '' 2024-09-16T11:44:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:44:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1.6 2024-09-16T11:44:09.633 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:09.631+0000 7f1fa11d1640 -1 reset not still connected to 0x565286653520 2024-09-16T11:44:09.634 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:09.631+0000 7f1fa11d1640 -1 reset not still connected to 0x56528ad55c70 2024-09-16T11:44:09.634 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:09.632+0000 7f1fa11d1640 -1 reset not still connected to 0x56528ad91ba0 2024-09-16T11:44:09.634 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:09.632+0000 7f1fa11d1640 -1 reset not still connected to 0x56528b091c70 2024-09-16T11:44:09.634 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:09.632+0000 7f1fa11d1640 -1 reset not still connected to 0x56528b389ad0 2024-09-16T11:44:09.634 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:09.632+0000 7f1fa11d1640 -1 reset not still connected to 0x56528bb9cb60 2024-09-16T11:44:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local log=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.status 2024-09-16T11:44:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T11:44:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T11:44:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local primary_snapshot_id=217 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 217 = 217 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:44:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 1 2024-09-16T11:44:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-16T11:44:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_50': 2024-09-16T11:44:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 128 MiB in 32 objects 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 2 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 12a0ae95515a 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.12a0ae95515a 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock, non-primary 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:35:55 2024 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:35:55 2024 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:35:55 2024 2024-09-16T11:44:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: protected: False 2024-09-16T11:44:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:44:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:44:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 52447688-b11b-4da0-8dc6-6ae9414561f1 2024-09-16T11:44:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: false 2024-09-16T11:44:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-16T11:44:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap_name=snap 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local ret=0 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local rmt_export=/tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.export 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local loc_export=/tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.export 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.export 2024-09-16T11:44:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.export 2024-09-16T11:44:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:44:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.export 2024-09-16T11:44:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:44:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cmp /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.export 2024-09-16T11:44:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -f /tmp/tmp.2dE0YG6s4q/cluster2-mirror-image_50.export /tmp/tmp.2dE0YG6s4q/cluster1-mirror-image_50.export 2024-09-16T11:44:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: delete many images' 2024-09-16T11:44:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:44:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:44:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:44:12 TEST: delete many images 2024-09-16T11:44:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:44:12 TEST: delete many images 2024-09-16T11:44:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 50 2024-09-16T11:44:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_1 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-16T11:44:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-16T11:44:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:44:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T11:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:44:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T11:44:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T11:44:12.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:12.900+0000 7ffbcfbeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed597380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:12.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:12.900+0000 7ffbd5bf7640 -1 rbd::mirror::ImageReplayer: 0x55a5ed3e5900 [3/dca53849-69bc-40ac-a731-5bbd18f66953] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_2 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-16T11:44:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-16T11:44:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:44:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T11:44:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:44:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T11:44:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T11:44:14.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:14.927+0000 7fe8de1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf0a39860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:15.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:14.927+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098bb80 [3/c33e6270-8531-45ef-bdc4-ecc6d15eb898] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:15.020 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:44:14.930+0000 7f1fa11d1640 -1 reset not still connected to 0x56528b389040 2024-09-16T11:44:15.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:15.125+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3c724e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:15.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:15.125+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098bb80 [3/c33e6270-8531-45ef-bdc4-ecc6d15eb898] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_3 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-16T11:44:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-16T11:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T11:44:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:44:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T11:44:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T11:44:17.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:17.132+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3c72820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:17.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:17.132+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098b180 [3/4b1d8b43-03c2-4e7d-bdc2-68e83df2c512] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_4 2024-09-16T11:44:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-16T11:44:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:44:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-16T11:44:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-16T11:44:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:44:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T11:44:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:44:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T11:44:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T11:44:18.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:18.936+0000 7fdf1ebea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561616b529c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:18.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:18.936+0000 7fdf25bf8640 -1 rbd::mirror::ImageReplayer: 0x5616136de500 [3/18f01d1c-b747-4fab-8897-4595d20eb919] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_5 2024-09-16T11:44:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-16T11:44:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:44:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-16T11:44:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-16T11:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T11:44:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:44:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T11:44:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T11:44:20.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:20.938+0000 7fdf1f3eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561616b52d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:20.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:20.938+0000 7fdf25bf8640 -1 rbd::mirror::ImageReplayer: 0x5616136de780 [3/735bbf09-b5b2-4a17-8414-9de64f208e63] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_6 2024-09-16T11:44:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-16T11:44:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:44:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-16T11:44:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-16T11:44:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:44:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T11:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:44:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T11:44:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T11:44:22.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:22.958+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99d4ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:22.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:22.958+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6253b80 [3/bd2d7a6e-4890-4807-895c-c7d327ef0b8d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_7 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-16T11:44:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-16T11:44:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:44:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T11:44:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:44:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T11:44:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T11:44:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:24.954+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99d5040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:24.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:24.954+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6252500 [3/215e6e09-36a7-4867-8fa8-8c5aab26f0d4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_8 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-16T11:44:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-16T11:44:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:44:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T11:44:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:44:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T11:44:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T11:44:26.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:26.961+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3c729c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:26.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:26.961+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098ac80 [3/8f02c93f-0a2c-444a-a327-af8d0dcd2b89] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_9 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-16T11:44:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-16T11:44:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:44:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T11:44:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:44:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T11:44:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T11:44:28.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:28.970+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3c72b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:28.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:28.970+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098b680 [3/ad8cd2a1-83b6-4d15-ba11-daa8d98ddb16] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_10 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-16T11:44:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-16T11:44:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:44:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T11:44:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:44:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T11:44:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T11:44:30.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:30.977+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d9992d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:30.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:30.977+0000 7f0980df8640 -1 rbd::mirror::ImageReplayer: 0x5591d952e780 [3/c727cb2e-9ff7-4230-b1cb-e1a3d78fbd0e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_11 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-16T11:44:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-16T11:44:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:44:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_11 2024-09-16T11:44:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:44:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-16T11:44:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-16T11:44:32.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:32.985+0000 7ffbcf3ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed5976c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:32.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:32.985+0000 7ffbd5bf7640 -1 rbd::mirror::ImageReplayer: 0x55a5ed4f7680 [3/d5fd87f4-333e-453f-8e09-9ee90d26a1c7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_12 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-16T11:44:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-16T11:44:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:44:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_12 2024-09-16T11:44:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:44:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-16T11:44:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-16T11:44:34.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:34.989+0000 7fdf1ebea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561616b53520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:34.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:34.989+0000 7fdf25bf8640 -1 rbd::mirror::ImageReplayer: 0x5616136df180 [3/a7a5a597-f597-46c5-bbc6-a5aec2f981b0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_13 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-16T11:44:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-16T11:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_13 2024-09-16T11:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:44:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-16T11:44:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-16T11:44:36.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:36.991+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d9992ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:36.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:36.991+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6252280 [3/f5a2a94d-7ff1-4c32-9b14-48a803629566] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_14 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:44:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-16T11:44:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-16T11:44:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:44:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_14 2024-09-16T11:44:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:44:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-16T11:44:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-16T11:44:39.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:39.015+0000 7ffbcfbeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed469d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:39.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:39.015+0000 7ffbd63f8640 -1 rbd::mirror::ImageReplayer: 0x55a5ed4f7400 [3/b897dc63-3368-46d2-ae8f-58605bd44840] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_15 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-16T11:44:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-16T11:44:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:44:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_15 2024-09-16T11:44:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:44:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-16T11:44:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-16T11:44:41.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:41.023+0000 7fe8de1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3c72d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:41.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:41.023+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098aa00 [3/8544ced6-19e9-448e-afa3-3117a33e55fb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_16 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:44:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-16T11:44:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-16T11:44:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:44:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_16 2024-09-16T11:44:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:44:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-16T11:44:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-16T11:44:42.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:42.802+0000 7ffbcf3ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed597860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:42.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:42.802+0000 7ffbd5bf7640 -1 rbd::mirror::ImageReplayer: 0x55a5ed2c4a00 [3/b478e757-78fd-4641-8d2d-50753c43edcb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_17 2024-09-16T11:44:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-16T11:44:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:44:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-16T11:44:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-16T11:44:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:44:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_17 2024-09-16T11:44:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:44:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-16T11:44:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-16T11:44:44.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:44.814+0000 7f097a5eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99d51e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:44.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:44.814+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d952e280 [3/a73d5ffa-b174-4dd8-82e4-e204e403ef6d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_18 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-16T11:44:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-16T11:44:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:44:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_18 2024-09-16T11:44:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:44:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-16T11:44:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-16T11:44:46.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:46.829+0000 7ffbcf3ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed288b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:46.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:46.829+0000 7ffbd63f8640 -1 rbd::mirror::ImageReplayer: 0x55a5ed485680 [3/b4ce7a6e-9d23-4e33-892a-2b51e7f4088f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_19 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:44:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-16T11:44:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-16T11:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_19 2024-09-16T11:44:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:44:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-16T11:44:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-16T11:44:48.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:48.833+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d9993040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:48.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:48.833+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6252780 [3/2d17f5e3-a0dd-42e5-91b8-084a792fce66] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_20 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-16T11:44:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-16T11:44:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:44:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_20 2024-09-16T11:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:44:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-16T11:44:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-16T11:44:50.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:50.837+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3bd2340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:50.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:44:50.837+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098af00 [3/42e976a5-afcc-4873-8194-2f268da32fe5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_21 2024-09-16T11:44:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-16T11:44:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:44:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-16T11:44:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-16T11:44:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:44:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_21 2024-09-16T11:44:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:44:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-16T11:44:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-16T11:44:52.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:52.843+0000 7f097a5eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d999dd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:52.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:44:52.843+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6252f00 [3/24a40005-617d-4202-82db-cfe123cbd527] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_22 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-16T11:44:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-16T11:44:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:44:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_22 2024-09-16T11:44:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:44:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-16T11:44:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-16T11:44:54.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:54.850+0000 7fdf1f3eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561616bf3ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:54.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:44:54.850+0000 7fdf25bf8640 -1 rbd::mirror::ImageReplayer: 0x5616169d6000 [3/c33407f3-8da2-4b7c-a242-5b063bcddddb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_23 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:44:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-16T11:44:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-16T11:44:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:44:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_23 2024-09-16T11:44:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:44:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-16T11:44:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-16T11:44:56.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:56.851+0000 7ffbcfbeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed4ee680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_24 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-16T11:44:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:44:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-16T11:44:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:44:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:44:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:44:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_24 2024-09-16T11:44:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:44:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-16T11:44:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-16T11:44:58.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:58.862+0000 7ffbcf3ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed2bbd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:44:58.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:44:58.862+0000 7ffbd63f8640 -1 rbd::mirror::ImageReplayer: 0x55a5ed485900 [3/d4a56ee3-400c-40d4-b957-ce8e33caa539] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:44:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_25 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:44:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-16T11:45:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-16T11:45:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:45:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_25 2024-09-16T11:45:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:45:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-16T11:45:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-16T11:45:00.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:00.865+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf0a39040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:00.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:00.865+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098a780 [3/72c9a059-c0ba-4ba8-a02d-26f894b5545c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_26 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-16T11:45:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-16T11:45:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:45:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_26 2024-09-16T11:45:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:45:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-16T11:45:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-16T11:45:02.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:02.891+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf0a39520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:02.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:02.891+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098a280 [3/05dd52ed-bc96-4184-b31a-04d2077723db] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_27 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-16T11:45:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-16T11:45:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:45:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_27 2024-09-16T11:45:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:45:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-16T11:45:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-16T11:45:04.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:04.876+0000 7ffbcf3ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed3dd380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:04.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:04.876+0000 7ffbd63f8640 -1 rbd::mirror::ImageReplayer: 0x55a5e9d85b80 [3/d7cd8c6a-1904-4edd-946f-03b287d93156] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_28 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-16T11:45:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-16T11:45:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:45:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_28 2024-09-16T11:45:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:45:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-16T11:45:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-16T11:45:06.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:06.902+0000 7ffbcf3ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed597a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:06.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:06.902+0000 7ffbd5bf7640 -1 rbd::mirror::ImageReplayer: 0x55a5ed2c4f00 [3/f168dae1-0384-473d-abfb-2c2c4e9f98dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:07.120 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6949276628839992) > weight (0.25) 2024-09-16T11:45:07.120 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T11:45:07.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T11:45:07.121 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T11:45:07.121 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T11:45:07.121 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:45:07.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:45:07.120+0000 7fb89b5f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 47929) UID: 1000 2024-09-16T11:45:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_29 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-16T11:45:08.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:45:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-16T11:45:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:45:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_29 2024-09-16T11:45:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:45:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-16T11:45:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-16T11:45:09.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:09.087+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d9993380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:09.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:09.087+0000 7f0980df8640 -1 rbd::mirror::ImageReplayer: 0x5591d6253900 [3/28f21500-e6e2-438c-ab9f-f105b1869d6b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:09.488 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:45:09.484+0000 7f9840b63640 -1 reset not still connected to 0x5603879ce000 2024-09-16T11:45:09.488 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:45:09.484+0000 7f9840b63640 -1 reset not still connected to 0x5603879ea820 2024-09-16T11:45:09.488 INFO:tasks.ceph.cluster1.osd.0.smithi001.stderr:2024-09-16T11:45:09.484+0000 7f9840b63640 -1 reset not still connected to 0x5603879eb860 2024-09-16T11:45:09.500 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:45:09.498+0000 7f1fa11d1640 -1 reset not still connected to 0x56528b388820 2024-09-16T11:45:09.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:45:09.512+0000 7fe7c9fde640 -1 rbd::mirror::ImageReplayer: 0x555d04fe3b80 [3/f10dfeca-d8b5-4422-b71a-0f86c6765b14] start: already running 2024-09-16T11:45:09.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:45:09.512+0000 7fe7c9fde640 -1 rbd::mirror::ImageReplayer: 0x555d01af9400 [3/6eadc152-f19b-4a5d-abbd-4c03a784ef9f] start: already running 2024-09-16T11:45:09.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:45:09.512+0000 7fe7c9fde640 -1 rbd::mirror::ImageReplayer: 0x555d01af8a00 [3/541cb1e3-d977-4c95-ba41-c949127ff0cd] start: already running 2024-09-16T11:45:09.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:45:09.516+0000 7f40501de640 -1 rbd::mirror::ImageReplayer: 0x55eecc66e500 [3/de17e228-9a77-43f1-bfec-27661cda333e] start: already running 2024-09-16T11:45:09.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:45:09.516+0000 7f40501de640 -1 rbd::mirror::ImageReplayer: 0x55eecc66ec80 [3/d6241982-1b6d-4d65-b374-39b9a477b51a] start: already running 2024-09-16T11:45:09.598 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:45:09.596+0000 7f1fa11d1640 -1 reset not still connected to 0x56528bde3ee0 2024-09-16T11:45:09.599 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:45:09.597+0000 7f1fa11d1640 -1 reset not still connected to 0x56528be4fc70 2024-09-16T11:45:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_30 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-16T11:45:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-16T11:45:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:45:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_30 2024-09-16T11:45:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:45:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-16T11:45:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-16T11:45:10.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:45:10.903+0000 7fdf1ebea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x561616bf3d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:10.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:45:10.903+0000 7fdf25bf8640 -1 rbd::mirror::ImageReplayer: 0x5616169d6280 [3/d6241982-1b6d-4d65-b374-39b9a477b51a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_31 2024-09-16T11:45:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-16T11:45:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:45:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-16T11:45:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-16T11:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_31 2024-09-16T11:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:45:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-16T11:45:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-16T11:45:12.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:12.911+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d96431e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:12.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:12.911+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6252c80 [3/57d96513-857d-403d-8a99-bc4b0fd3e473] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:13.223 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T11:45:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:45:13.223 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:45:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:45:13.227 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-16T11:45:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_32 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:45:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-16T11:45:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-16T11:45:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:45:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_32 2024-09-16T11:45:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:45:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-16T11:45:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-16T11:45:14.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:14.918+0000 7fe8de1ea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf0a39380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:14.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:14.918+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098b400 [3/905fd472-3e29-4634-b6a3-059626393330] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:15.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:15.120+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3e02000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:15.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:15.120+0000 7fe8e51f8640 -1 rbd::mirror::ImageReplayer: 0x55dbf098b400 [3/905fd472-3e29-4634-b6a3-059626393330] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_33 2024-09-16T11:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-16T11:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-16T11:45:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-16T11:45:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:45:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_33 2024-09-16T11:45:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:45:16.790 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-16T11:45:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-16T11:45:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:45:16.887+0000 7fe7d67f7640 -1 rbd::mirror::ImageMap: 0x555d057fe6c0 handle_peer_ack_remove: failed to remove global_image_id=d843e793-ea22-4d9f-a27b-0c60b2326370 2024-09-16T11:45:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_34 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:45:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-16T11:45:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-16T11:45:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:45:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_34 2024-09-16T11:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:45:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-16T11:45:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-16T11:45:18.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:18.956+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99924e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:18.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:18.956+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d6252a00 [3/6abecb8f-c955-47d0-8423-8d9b6c3ccf30] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:19.003 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.44607790218766374) > weight (0.25) 2024-09-16T11:45:19.003 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.601300510162346) > weight (0.25) 2024-09-16T11:45:19.003 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T11:45:19.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T11:45:19.004 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-16T11:45:19.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:45:19.003+0000 7fdf263f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 60164) UID: 1000 2024-09-16T11:45:19.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_35 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-16T11:45:20.278 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:45:20.276+0000 7f0000d63640 -1 reset not still connected to 0x55b11fb30340 2024-09-16T11:45:20.278 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:45:20.276+0000 7f0000d63640 -1 reset not still connected to 0x55b124355c70 2024-09-16T11:45:20.278 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:45:20.276+0000 7f0000d63640 -1 reset not still connected to 0x55b1243e5a00 2024-09-16T11:45:20.279 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:45:20.276+0000 7f0000d63640 -1 reset not still connected to 0x55b125b2aea0 2024-09-16T11:45:20.279 INFO:tasks.ceph.cluster1.osd.1.smithi001.stderr:2024-09-16T11:45:20.276+0000 7f0000d63640 -1 reset not still connected to 0x55b125b2af70 2024-09-16T11:45:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-16T11:45:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:45:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_35 2024-09-16T11:45:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:45:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-16T11:45:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-16T11:45:20.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:20.955+0000 7f0980df8640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=22d639e5-b167-4825-bd14-2050baefdab4 2024-09-16T11:45:21.005 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T11:45:21.005 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:45:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:21.006 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T11:45:21.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T11:45:21.006 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T11:45:21.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T11:45:21.009 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-16T11:45:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_36 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-16T11:45:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-16T11:45:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_36 2024-09-16T11:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:45:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-16T11:45:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-16T11:45:22.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:22.966+0000 7f0980df8640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=5caa0b20-60a7-472a-a287-3766f979eba6 2024-09-16T11:45:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_37 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-16T11:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-16T11:45:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:45:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_37 2024-09-16T11:45:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:45:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-16T11:45:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-16T11:45:24.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:45:24.933+0000 7fe7d67f7640 -1 rbd::mirror::ImageMap: 0x555d057fe6c0 handle_peer_ack_remove: failed to remove global_image_id=eb944d6a-f8cb-4ae2-a094-e3ad1747c24a 2024-09-16T11:45:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_38 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-16T11:45:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-16T11:45:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:45:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_38 2024-09-16T11:45:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:45:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-16T11:45:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-16T11:45:26.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:45:26.937+0000 7fe7d6ff8640 -1 rbd::mirror::ImageMap: 0x555d057fe6c0 handle_peer_ack_remove: failed to remove global_image_id=7445f68a-3ed3-43b0-b798-83462bd9fca7 2024-09-16T11:45:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_39 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-16T11:45:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-16T11:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_39 2024-09-16T11:45:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:45:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-16T11:45:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-16T11:45:28.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:28.992+0000 7ffbcfbeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed38d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:28.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:28.992+0000 7ffbd63f8640 -1 rbd::mirror::ImageReplayer: 0x55a5e9d84280 [3/e54108d2-02f3-4f44-b0c9-f084887582f4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_40 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-16T11:45:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-16T11:45:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:45:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_40 2024-09-16T11:45:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:45:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-16T11:45:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-16T11:45:30.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:30.990+0000 7f097a5eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d9928ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:30.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:30.990+0000 7f0980df8640 -1 rbd::mirror::ImageReplayer: 0x5591d952e500 [3/de17e228-9a77-43f1-bfec-27661cda333e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_41 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-16T11:45:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-16T11:45:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:45:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_41 2024-09-16T11:45:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:45:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-16T11:45:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-16T11:45:32.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:32.815+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d9929520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:32.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:32.815+0000 7f0980df8640 -1 rbd::mirror::ImageReplayer: 0x5591d6253680 [3/316bb7bf-0be4-4d21-b8d9-199ffb446933] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_42 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-16T11:45:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-16T11:45:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:45:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_42 2024-09-16T11:45:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:45:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-16T11:45:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-16T11:45:34.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:34.829+0000 7ffbcfbeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed597d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:34.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:34.829+0000 7ffbd5bf7640 -1 rbd::mirror::ImageReplayer: 0x55a5ed4f7900 [3/f10dfeca-d8b5-4422-b71a-0f86c6765b14] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_43 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-16T11:45:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-16T11:45:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:45:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_43 2024-09-16T11:45:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:45:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-16T11:45:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-16T11:45:36.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:36.839+0000 7f09805f7640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=6eadc152-f19b-4a5d-abbd-4c03a784ef9f 2024-09-16T11:45:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_44 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-16T11:45:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-16T11:45:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:45:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_44 2024-09-16T11:45:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:45:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-16T11:45:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-16T11:45:38.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:38.845+0000 7f0980df8640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=0aab3740-d202-4c69-8377-5da1f823faf0 2024-09-16T11:45:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_45 2024-09-16T11:45:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-16T11:45:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:45:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-16T11:45:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-16T11:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_45 2024-09-16T11:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-16T11:45:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-16T11:45:40.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:40.858+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99ca1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:40.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:40.858+0000 7f09805f7640 -1 rbd::mirror::ImageReplayer: 0x5591d952e000 [3/4de1128c-eeed-43f2-8404-66dfce23f601] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:41.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:41.544+0000 7f0980df8640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=7445f68a-3ed3-43b0-b798-83462bd9fca7 2024-09-16T11:45:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_46 2024-09-16T11:45:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-16T11:45:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:45:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-16T11:45:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-16T11:45:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:45:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_46 2024-09-16T11:45:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:45:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-16T11:45:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-16T11:45:42.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:45:42.864+0000 7f0980df8640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=541cb1e3-d977-4c95-ba41-c949127ff0cd 2024-09-16T11:45:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_47 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-16T11:45:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-16T11:45:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:45:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_47 2024-09-16T11:45:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:45:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-16T11:45:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-16T11:45:44.839 INFO:tasks.ceph.cluster1.osd.2.smithi001.stderr:2024-09-16T11:45:44.837+0000 7f1fa11d1640 -1 reset not still connected to 0x56528b388820 2024-09-16T11:45:44.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:45:44.862+0000 7fafc55eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564bcb26ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:44.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:45:44.862+0000 7fafcbdf8640 -1 rbd::mirror::ImageReplayer: 0x564bc7ffe280 [3/a52e8873-193f-487c-b67c-f482a83709db] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_48 2024-09-16T11:45:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-16T11:45:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-16T11:45:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-16T11:45:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:45:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_48 2024-09-16T11:45:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:45:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-16T11:45:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-16T11:45:46.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:46.888+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3dff6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:46.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:46.888+0000 7fe8e49f7640 -1 rbd::mirror::ImageReplayer: 0x55dbf3cda780 [3/67ab9ab4-c46d-466c-848f-7cc71d56e268] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_49 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-16T11:45:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-16T11:45:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:45:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_49 2024-09-16T11:45:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:45:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-16T11:45:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-16T11:45:48.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:48.890+0000 7fe8de9eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dbf3dffa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:48.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:45:48.890+0000 7fe8e49f7640 -1 rbd::mirror::ImageReplayer: 0x55dbf098a500 [3/1040264d-d1a6-45fc-81f5-3534645345d5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_50 2024-09-16T11:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-16T11:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local snap=snap 2024-09-16T11:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-16T11:45:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing snap: 100% complete...done. 2024-09-16T11:45:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-16T11:45:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:45:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:45:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:45:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_50 2024-09-16T11:45:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:45:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:45:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-16T11:45:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-16T11:45:50.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:50.892+0000 7ffbcfbeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a5ed3efba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:45:50.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:45:50.892+0000 7ffbd5bf7640 -1 rbd::mirror::ImageReplayer: 0x55a5ed4f6a00 [3/52447688-b11b-4da0-8dc6-6ae9414561f1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:45:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:45:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-16T11:45:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:45:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:45:51 TEST: image deletions should propagate 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:45:51 TEST: image deletions should propagate 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_count=0 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local count 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local last_count=0 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ true 2024-09-16T11:45:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 40 2024-09-16T11:45:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:45:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 1 -ne 1 2024-09-16T11:45:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T11:45:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'images: ' 2024-09-16T11:45:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 2 2024-09-16T11:45:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ count=0 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state 2024-09-16T11:45:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 40 2024-09-16T11:45:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in `seq 1 40` 2024-09-16T11:45:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 1 -ne 1 2024-09-16T11:45:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T11:45:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep 'image health:' 2024-09-16T11:45:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cut '-d ' -f 3 2024-09-16T11:45:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T11:45:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ state=OK 2024-09-16T11:45:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test OK = ERROR 2024-09-16T11:45:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test OK = OK 2024-09-16T11:45:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 50 2024-09-16T11:45:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_1 2024-09-16T11:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-16T11:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T11:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_1 2024-09-16T11:45:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T11:45:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T11:45:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-16T11:45:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:45:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_1 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T11:45:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_2 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-16T11:45:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_2 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T11:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T11:45:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_2 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T11:45:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T11:45:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_3 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_3 2024-09-16T11:45:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T11:45:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T11:45:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-16T11:45:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_3 2024-09-16T11:45:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T11:45:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T11:45:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_4 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T11:45:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_4 2024-09-16T11:45:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T11:45:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T11:45:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-16T11:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_4 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T11:45:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_5 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T11:45:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_5 2024-09-16T11:45:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T11:45:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T11:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_5 2024-09-16T11:45:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T11:45:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_6 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-16T11:45:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_6 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T11:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T11:45:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-16T11:45:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_6 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T11:45:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_7 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-16T11:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_7 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T11:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T11:45:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-16T11:45:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_7 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T11:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T11:45:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_8 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T11:45:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_8 2024-09-16T11:45:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T11:45:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T11:45:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_8 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T11:45:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_9 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:45:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_9 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T11:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T11:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T11:45:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_9 2024-09-16T11:45:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T11:45:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T11:45:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_10 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_10 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T11:45:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T11:45:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-16T11:45:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_10 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T11:45:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T11:45:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_11 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_11 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_11 2024-09-16T11:45:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-16T11:45:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-16T11:45:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_11 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_11 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_11 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-16T11:45:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_12 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_12 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_12 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-16T11:45:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-16T11:45:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_12 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_12 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_12 2024-09-16T11:45:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-16T11:45:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-16T11:45:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_13 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_13 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_13 2024-09-16T11:45:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-16T11:45:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-16T11:45:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-16T11:45:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_13 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_13 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_13 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-16T11:45:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-16T11:45:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_14 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_14 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_14 2024-09-16T11:45:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-16T11:45:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-16T11:45:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-16T11:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_14 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_14 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_14 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-16T11:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-16T11:45:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_15 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_15 2024-09-16T11:45:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_15 2024-09-16T11:45:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-16T11:45:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-16T11:45:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_15 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_15 2024-09-16T11:45:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_15 2024-09-16T11:45:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-16T11:45:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_16 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_16 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_16 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-16T11:45:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-16T11:45:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-16T11:45:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_16 2024-09-16T11:45:54.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_16 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_16 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-16T11:45:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_17 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-16T11:45:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_17 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_17 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-16T11:45:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-16T11:45:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-16T11:45:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_17 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_17 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_17 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-16T11:45:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-16T11:45:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_18 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_18 2024-09-16T11:45:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_18 2024-09-16T11:45:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-16T11:45:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-16T11:45:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_18 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_18 2024-09-16T11:45:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_18 2024-09-16T11:45:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-16T11:45:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-16T11:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_19 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_19 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_19 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-16T11:45:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-16T11:45:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_19 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_19 2024-09-16T11:45:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_19 2024-09-16T11:45:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-16T11:45:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-16T11:45:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_20 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_20 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_20 2024-09-16T11:45:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-16T11:45:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-16T11:45:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_20 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_20 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_20 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-16T11:45:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_21 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_21 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_21 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-16T11:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-16T11:45:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-16T11:45:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_21 2024-09-16T11:45:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_21 2024-09-16T11:45:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_21 2024-09-16T11:45:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-16T11:45:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-16T11:45:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_22 2024-09-16T11:45:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-16T11:45:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:45:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_22 2024-09-16T11:45:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_22 2024-09-16T11:45:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-16T11:45:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-16T11:45:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-16T11:45:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_22 2024-09-16T11:45:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_22 2024-09-16T11:45:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_22 2024-09-16T11:45:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-16T11:45:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-16T11:45:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_23 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_23 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_23 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-16T11:45:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-16T11:45:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-16T11:45:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_23 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_23 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_23 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-16T11:45:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_24 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_24 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_24 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-16T11:45:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-16T11:45:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-16T11:45:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_24 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_24 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_24 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-16T11:45:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_25 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:45:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_25 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_25 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-16T11:45:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-16T11:45:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_25 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_25 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_25 2024-09-16T11:45:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-16T11:45:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-16T11:45:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_26 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_26 2024-09-16T11:45:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_26 2024-09-16T11:45:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-16T11:45:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-16T11:45:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_26 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_26 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_26 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-16T11:45:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-16T11:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_27 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_27 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_27 2024-09-16T11:45:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-16T11:45:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-16T11:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_27 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_27 2024-09-16T11:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_27 2024-09-16T11:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-16T11:45:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_28 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:45:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_28 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_28 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-16T11:45:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-16T11:45:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_28 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_28 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_28 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-16T11:45:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-16T11:45:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_29 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_29 2024-09-16T11:45:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_29 2024-09-16T11:45:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-16T11:45:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-16T11:45:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_29 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_29 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_29 2024-09-16T11:45:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-16T11:45:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-16T11:45:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_30 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_30 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_30 2024-09-16T11:45:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-16T11:45:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-16T11:45:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_30 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_30 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_30 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-16T11:45:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-16T11:45:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_31 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_31 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_31 2024-09-16T11:45:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-16T11:45:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-16T11:45:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_31 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_31 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_31 2024-09-16T11:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-16T11:45:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-16T11:45:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_32 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_32 2024-09-16T11:45:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_32 2024-09-16T11:45:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-16T11:45:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-16T11:45:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-16T11:45:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_32 2024-09-16T11:45:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_32 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_32 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-16T11:45:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-16T11:45:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_33 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_33 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_33 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-16T11:45:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-16T11:45:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_33 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_33 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_33 2024-09-16T11:45:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-16T11:45:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-16T11:45:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_34 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_34 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_34 2024-09-16T11:45:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-16T11:45:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-16T11:45:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-16T11:45:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_34 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_34 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_34 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-16T11:45:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-16T11:45:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_35 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_35 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_35 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-16T11:45:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-16T11:45:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-16T11:45:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_35 2024-09-16T11:45:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_35 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_35 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-16T11:45:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_36 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:45:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_36 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_36 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-16T11:45:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-16T11:45:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-16T11:45:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_36 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_36 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_36 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-16T11:45:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-16T11:45:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_37 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_37 2024-09-16T11:45:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_37 2024-09-16T11:45:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-16T11:45:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-16T11:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_37 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_37 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_37 2024-09-16T11:45:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-16T11:45:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_38 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_38 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_38 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-16T11:45:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-16T11:45:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-16T11:45:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_38 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_38 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_38 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-16T11:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_39 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_39 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_39 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-16T11:45:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-16T11:45:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-16T11:45:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_39 2024-09-16T11:45:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_39 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_39 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-16T11:45:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-16T11:45:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_40 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_40 2024-09-16T11:45:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_40 2024-09-16T11:45:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-16T11:45:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-16T11:45:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-16T11:45:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_40 2024-09-16T11:45:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_40 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_40 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-16T11:45:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-16T11:45:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_41 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_41 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_41 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-16T11:45:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-16T11:45:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_41 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_41 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_41 2024-09-16T11:45:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-16T11:45:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-16T11:45:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_42 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_42 2024-09-16T11:45:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_42 2024-09-16T11:45:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-16T11:45:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-16T11:45:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_42 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_42 2024-09-16T11:45:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_42 2024-09-16T11:45:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-16T11:45:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_43 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_43 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_43 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-16T11:45:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-16T11:45:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_43 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_43 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_43 2024-09-16T11:45:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-16T11:45:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-16T11:45:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_44 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_44 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_44 2024-09-16T11:45:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-16T11:45:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-16T11:45:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-16T11:45:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_44 2024-09-16T11:45:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_44 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_44 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-16T11:45:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-16T11:45:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_45 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_45 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_45 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-16T11:45:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-16T11:45:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:45:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_45 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_45 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_45 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-16T11:45:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-16T11:45:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:45:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:45:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_46 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_46 2024-09-16T11:45:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:45:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:45:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_46 2024-09-16T11:45:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-16T11:45:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:45:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-16T11:45:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:45:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:45:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_46 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_46 2024-09-16T11:46:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_46 2024-09-16T11:46:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-16T11:46:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-16T11:46:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_47 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_47 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_47 2024-09-16T11:46:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-16T11:46:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-16T11:46:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:46:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_47 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_47 2024-09-16T11:46:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_47 2024-09-16T11:46:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-16T11:46:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-16T11:46:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_48 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:46:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_48 2024-09-16T11:46:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_48 2024-09-16T11:46:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-16T11:46:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-16T11:46:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:46:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-16T11:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_48 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_48 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_48 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-16T11:46:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_49 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_49 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_49 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-16T11:46:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-16T11:46:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:46:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-16T11:46:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_49 2024-09-16T11:46:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_49 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_49 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-16T11:46:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_50 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_50 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_50 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-16T11:46:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-16T11:46:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:46:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_50 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_50 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_50 2024-09-16T11:46:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-16T11:46:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-16T11:46:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-16T11:46:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:46:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:46:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:46:00 TEST: delete images during bootstrap 2024-09-16T11:46:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:46:00 TEST: delete images during bootstrap 2024-09-16T11:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-16T11:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=image 2024-09-16T11:46:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-16T11:46:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd: mirroring is already configured for image mode 2024-09-16T11:46:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-16T11:46:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=image 2024-09-16T11:46:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-16T11:46:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd: mirroring is already configured for image mode 2024-09-16T11:46:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ start_mirror cluster1 2024-09-16T11:46:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance 2024-09-16T11:46:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-16T11:46:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local val=cluster1 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster_var_name=cluster 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local instance_var_name=instance 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cluster=cluster1 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=cluster1 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ instance=0 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval cluster=cluster1 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ cluster=cluster1 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ eval instance=0 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ instance=0 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=test 2024-09-16T11:46:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 1 10 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_1 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:46:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:46:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2024-09-16T11:46:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-16T11:46:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:46:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-16T11:46:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:46:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_1': 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 192fab6d227b 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.192fab6d227b 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:46:01 2024 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:46:01 2024 2024-09-16T11:46:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:46:01 2024 2024-09-16T11:46:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:46:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:46:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: d85ad0bd-7ee5-4637-974b-18591cb72e86 2024-09-16T11:46:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:46:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-16T11:46:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:46:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:46:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:46:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:46:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 78509056 bytes 2024-09-16T11:46:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 97487872 bytes 2024-09-16T11:46:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 100460544 bytes 2024-09-16T11:46:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 119684096 bytes 2024-09-16T11:46:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 138099712 bytes 2024-09-16T11:46:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 171271168 bytes 2024-09-16T11:46:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 200964096 bytes 2024-09-16T11:46:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 268878848 bytes 2024-09-16T11:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 293055488 bytes 2024-09-16T11:46:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 328354816 bytes 2024-09-16T11:46:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 358278144 bytes 2024-09-16T11:46:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 408653824 bytes 2024-09-16T11:46:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 433973248 bytes 2024-09-16T11:46:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 489927680 bytes 2024-09-16T11:46:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 521067520 bytes 2024-09-16T11:46:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 586828800 bytes 2024-09-16T11:46:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 635376640 bytes 2024-09-16T11:46:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 674477056 bytes 2024-09-16T11:46:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 781304832 bytes 2024-09-16T11:46:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 815998976 bytes 2024-09-16T11:46:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 901254144 bytes 2024-09-16T11:46:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 920161280 bytes 2024-09-16T11:46:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 936792064 bytes 2024-09-16T11:46:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 966349824 bytes 2024-09-16T11:46:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1011603456 bytes 2024-09-16T11:46:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1034580992 bytes 2024-09-16T11:46:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1035180032 bytes 2024-09-16T11:46:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1137074176 bytes 2024-09-16T11:46:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1181570048 bytes 2024-09-16T11:46:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1209592832 bytes 2024-09-16T11:46:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1318226944 bytes 2024-09-16T11:46:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1340331008 bytes 2024-09-16T11:46:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1413953536 bytes 2024-09-16T11:46:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1451934720 bytes 2024-09-16T11:46:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1463616512 bytes 2024-09-16T11:46:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1468906496 bytes 2024-09-16T11:46:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1511811072 bytes 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_1 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T11:46:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=1509cc9f7ee 2024-09-16T11:46:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_1 present 1509cc9f7ee 2024-09-16T11:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=1509cc9f7ee 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_1 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T11:46:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=1509cc9f7ee 2024-09-16T11:46:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1509cc9f7ee 2024-09-16T11:46:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 1509cc9f7ee -o 1509cc9f7ee = 1509cc9f7ee 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-16T11:46:08.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:46:08.034+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d0532a000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2024-09-16T11:46:08.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:46:08.034+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d0532a000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2024-09-16T11:46:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-16T11:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T11:46:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_1 2024-09-16T11:46:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T11:46:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T11:46:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:46:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_1 2024-09-16T11:46:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T11:46:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T11:46:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-16T11:46:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-16T11:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-16T11:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:46:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:46:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T11:46:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_1 2024-09-16T11:46:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T11:46:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T11:46:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:46:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_2 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:46:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:46:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:46:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2024-09-16T11:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-16T11:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-16T11:46:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:46:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_2': 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1947de24411a 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1947de24411a 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:46:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:46:37 2024 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:46:37 2024 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:46:37 2024 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: cd2c332c-c624-4624-8f87-5506e0924f3c 2024-09-16T11:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:46:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-16T11:46:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:46:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=12 2024-09-16T11:46:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:46:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:46:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 73316352 bytes 2024-09-16T11:46:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 125573120 bytes 2024-09-16T11:46:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 131599360 bytes 2024-09-16T11:46:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 143677440 bytes 2024-09-16T11:46:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 197671936 bytes 2024-09-16T11:46:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 208214016 bytes 2024-09-16T11:46:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 321669120 bytes 2024-09-16T11:46:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 384815104 bytes 2024-09-16T11:46:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 436280320 bytes 2024-09-16T11:46:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 498847744 bytes 2024-09-16T11:46:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 522447872 bytes 2024-09-16T11:46:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 554378240 bytes 2024-09-16T11:46:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 637677568 bytes 2024-09-16T11:46:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 718158848 bytes 2024-09-16T11:46:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 729034752 bytes 2024-09-16T11:46:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 729648128 bytes 2024-09-16T11:46:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 784761856 bytes 2024-09-16T11:46:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 828364800 bytes 2024-09-16T11:46:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 841360384 bytes 2024-09-16T11:46:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 865160192 bytes 2024-09-16T11:46:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 886832128 bytes 2024-09-16T11:46:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 907310080 bytes 2024-09-16T11:46:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 944382976 bytes 2024-09-16T11:46:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 966171648 bytes 2024-09-16T11:46:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1001123840 bytes 2024-09-16T11:46:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1055002624 bytes 2024-09-16T11:46:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1166988288 bytes 2024-09-16T11:46:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1225448448 bytes 2024-09-16T11:46:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1246044160 bytes 2024-09-16T11:46:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1316962304 bytes 2024-09-16T11:46:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1367314432 bytes 2024-09-16T11:46:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1391047680 bytes 2024-09-16T11:46:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1437702144 bytes 2024-09-16T11:46:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1440736256 bytes 2024-09-16T11:46:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1526665216 bytes 2024-09-16T11:46:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1564705792 bytes 2024-09-16T11:46:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1596377088 bytes 2024-09-16T11:46:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1631027200 bytes 2024-09-16T11:46:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1669770240 bytes 2024-09-16T11:46:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1670356992 bytes 2024-09-16T11:46:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1706703872 bytes 2024-09-16T11:46:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1724766208 bytes 2024-09-16T11:46:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1725381632 bytes 2024-09-16T11:46:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1742314496 bytes 2024-09-16T11:46:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1830937600 bytes 2024-09-16T11:46:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1871756288 bytes 2024-09-16T11:46:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1877713920 bytes 2024-09-16T11:46:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1960365056 bytes 2024-09-16T11:46:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1981213696 bytes 2024-09-16T11:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2008154112 bytes 2024-09-16T11:46:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2013627392 bytes 2024-09-16T11:46:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2032889856 bytes 2024-09-16T11:46:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2103067648 bytes 2024-09-16T11:46:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2169016320 bytes 2024-09-16T11:46:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2257009664 bytes 2024-09-16T11:46:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2388965376 bytes 2024-09-16T11:46:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2448566272 bytes 2024-09-16T11:46:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2473424896 bytes 2024-09-16T11:46:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2483311616 bytes 2024-09-16T11:46:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2511749120 bytes 2024-09-16T11:46:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2590752768 bytes 2024-09-16T11:46:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2650980352 bytes 2024-09-16T11:46:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2662068224 bytes 2024-09-16T11:46:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2664825856 bytes 2024-09-16T11:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2703620096 bytes 2024-09-16T11:46:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2789296128 bytes 2024-09-16T11:46:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2859712512 bytes 2024-09-16T11:46:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2899876864 bytes 2024-09-16T11:46:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2958859264 bytes 2024-09-16T11:46:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:46:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_2 2024-09-16T11:46:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T11:46:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=150997ddd17b 2024-09-16T11:46:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_2 present 150997ddd17b 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=150997ddd17b 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_2 2024-09-16T11:46:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T11:46:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=150997ddd17b 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 150997ddd17b 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 150997ddd17b -o 150997ddd17b = 150997ddd17b 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-16T11:46:51.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:46:51.784+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d0502e400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2024-09-16T11:46:51.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:46:51.784+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d0502e400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2024-09-16T11:46:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_2 2024-09-16T11:46:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T11:46:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T11:46:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:46:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:46:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:46:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-16T11:46:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:46:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:46:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_2 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T11:46:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:46:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-16T11:46:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-16T11:46:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:46:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_2 2024-09-16T11:46:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T11:46:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T11:47:09.227 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8232240483051798) > weight (0.25) 2024-09-16T11:47:09.227 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4011765209389878) > weight (0.25) 2024-09-16T11:47:09.227 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6756433540007715) > weight (0.25) 2024-09-16T11:47:09.228 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.37973267550682044) > weight (0.25) 2024-09-16T11:47:09.228 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-16T11:47:19.009 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6202760042272755) > weight (0.25) 2024-09-16T11:47:19.009 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T11:47:19.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T11:47:19.010 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T11:47:19.010 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T11:47:19.010 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:47:19.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:47:19.008+0000 7ffbd6bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 46452) UID: 1000 2024-09-16T11:47:19.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:47:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:47:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:47:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:47:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_3 2024-09-16T11:47:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-16T11:47:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:47:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2024-09-16T11:47:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-16T11:47:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:47:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-16T11:47:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:47:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-16T11:47:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_3': 2024-09-16T11:47:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 195fd3a96a40 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.195fd3a96a40 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:47:20 2024 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:47:20 2024 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:47:20 2024 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:47:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: b57784e9-2291-413a-814c-df92f1484256 2024-09-16T11:47:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:47:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-16T11:47:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:47:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=14 2024-09-16T11:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:47:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:47:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 66064384 bytes 2024-09-16T11:47:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 133268480 bytes 2024-09-16T11:47:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 169729024 bytes 2024-09-16T11:47:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 197338112 bytes 2024-09-16T11:47:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 239374336 bytes 2024-09-16T11:47:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 297498624 bytes 2024-09-16T11:47:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 332219392 bytes 2024-09-16T11:47:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 376010752 bytes 2024-09-16T11:47:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 409574400 bytes 2024-09-16T11:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 444633088 bytes 2024-09-16T11:47:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 503441408 bytes 2024-09-16T11:47:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 524980224 bytes 2024-09-16T11:47:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 564320256 bytes 2024-09-16T11:47:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 612725760 bytes 2024-09-16T11:47:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 622605312 bytes 2024-09-16T11:47:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 658777088 bytes 2024-09-16T11:47:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 706098176 bytes 2024-09-16T11:47:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 749296640 bytes 2024-09-16T11:47:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 774320128 bytes 2024-09-16T11:47:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 815875072 bytes 2024-09-16T11:47:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 822018048 bytes 2024-09-16T11:47:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 871940096 bytes 2024-09-16T11:47:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 922058752 bytes 2024-09-16T11:47:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 994132992 bytes 2024-09-16T11:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1038092288 bytes 2024-09-16T11:47:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:47:25.111 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T11:47:25.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T11:47:25.111 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T11:47:25.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T11:47:25.114 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-16T11:47:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1104737280 bytes 2024-09-16T11:47:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1129631744 bytes 2024-09-16T11:47:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1133810688 bytes 2024-09-16T11:47:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1208153088 bytes 2024-09-16T11:47:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1327529984 bytes 2024-09-16T11:47:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1360480256 bytes 2024-09-16T11:47:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1390738432 bytes 2024-09-16T11:47:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1428344832 bytes 2024-09-16T11:47:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1462729728 bytes 2024-09-16T11:47:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1506636800 bytes 2024-09-16T11:47:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1573093376 bytes 2024-09-16T11:47:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1587265536 bytes 2024-09-16T11:47:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1678870528 bytes 2024-09-16T11:47:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1703187456 bytes 2024-09-16T11:47:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1778306048 bytes 2024-09-16T11:47:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1849910272 bytes 2024-09-16T11:47:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1873084416 bytes 2024-09-16T11:47:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1910394880 bytes 2024-09-16T11:47:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1951002624 bytes 2024-09-16T11:47:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2001340416 bytes 2024-09-16T11:47:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2030738432 bytes 2024-09-16T11:47:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2053904384 bytes 2024-09-16T11:47:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2111667200 bytes 2024-09-16T11:47:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2192286720 bytes 2024-09-16T11:47:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2235831296 bytes 2024-09-16T11:47:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2349776896 bytes 2024-09-16T11:47:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2357228544 bytes 2024-09-16T11:47:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2384411648 bytes 2024-09-16T11:47:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2456863744 bytes 2024-09-16T11:47:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2504188928 bytes 2024-09-16T11:47:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2543248384 bytes 2024-09-16T11:47:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2579050496 bytes 2024-09-16T11:47:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2595067904 bytes 2024-09-16T11:47:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2618125312 bytes 2024-09-16T11:47:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2618758144 bytes 2024-09-16T11:47:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2625071104 bytes 2024-09-16T11:47:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2684667904 bytes 2024-09-16T11:47:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2686262272 bytes 2024-09-16T11:47:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2749063168 bytes 2024-09-16T11:47:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2784801792 bytes 2024-09-16T11:47:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2857950208 bytes 2024-09-16T11:47:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2860922880 bytes 2024-09-16T11:47:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2878781440 bytes 2024-09-16T11:47:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2923983872 bytes 2024-09-16T11:47:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2964070400 bytes 2024-09-16T11:47:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 3049421824 bytes 2024-09-16T11:47:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 3085632512 bytes 2024-09-16T11:47:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 3117851648 bytes 2024-09-16T11:47:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 3145410560 bytes 2024-09-16T11:47:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3189270528 bytes 2024-09-16T11:47:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3254816768 bytes 2024-09-16T11:47:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3344364544 bytes 2024-09-16T11:47:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3399067648 bytes 2024-09-16T11:47:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3477584896 bytes 2024-09-16T11:47:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8000 IOs, 3516618752 bytes 2024-09-16T11:47:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:8100 IOs, 3559791616 bytes 2024-09-16T11:47:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T11:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_3 2024-09-16T11:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T11:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:47:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=1509c2b44cf8 2024-09-16T11:47:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:47:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:47:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_3 present 1509c2b44cf8 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=1509c2b44cf8 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_3 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T11:47:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=1509c2b44cf8 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1509c2b44cf8 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 1509c2b44cf8 -o 1509c2b44cf8 = 1509c2b44cf8 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-16T11:47:36.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:47:36.200+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d05031c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2024-09-16T11:47:36.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:47:36.200+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d05031c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2024-09-16T11:47:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:47:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:47:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T11:47:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_3 2024-09-16T11:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T11:47:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:47:41.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:47:41.228+0000 7f0980df8640 -1 rbd::mirror::ImageMap: 0x5591d937afc0 handle_peer_ack_remove: failed to remove global_image_id=b57784e9-2291-413a-814c-df92f1484256 2024-09-16T11:47:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T11:47:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:47:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:47:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:47:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:47:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:47:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_3 2024-09-16T11:47:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T11:47:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:47:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T11:47:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-16T11:47:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-16T11:47:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:47:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:47:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T11:47:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:47:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:47:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_3 2024-09-16T11:47:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T11:47:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T11:48:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:48:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:48:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_4 2024-09-16T11:48:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-16T11:48:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2024-09-16T11:48:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-16T11:48:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:48:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-16T11:48:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:48:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_4': 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 19804764acde 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.19804764acde 2024-09-16T11:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:48:06 2024 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:48:06 2024 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:48:06 2024 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 1788ecf7-05cd-419f-a3a1-739f93adb69e 2024-09-16T11:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:48:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-16T11:48:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:48:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:48:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:48:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:48:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 17443840 bytes 2024-09-16T11:48:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 34442240 bytes 2024-09-16T11:48:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 89314304 bytes 2024-09-16T11:48:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 159577088 bytes 2024-09-16T11:48:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 309563392 bytes 2024-09-16T11:48:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 323813376 bytes 2024-09-16T11:48:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 342274048 bytes 2024-09-16T11:48:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 418515968 bytes 2024-09-16T11:48:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 470730752 bytes 2024-09-16T11:48:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 483489792 bytes 2024-09-16T11:48:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 512882688 bytes 2024-09-16T11:48:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 573595648 bytes 2024-09-16T11:48:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 665118720 bytes 2024-09-16T11:48:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 684650496 bytes 2024-09-16T11:48:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 799700992 bytes 2024-09-16T11:48:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 855777280 bytes 2024-09-16T11:48:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 907962368 bytes 2024-09-16T11:48:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 957022208 bytes 2024-09-16T11:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1015878656 bytes 2024-09-16T11:48:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1029957632 bytes 2024-09-16T11:48:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1057164288 bytes 2024-09-16T11:48:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1128262656 bytes 2024-09-16T11:48:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1182026752 bytes 2024-09-16T11:48:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1208566784 bytes 2024-09-16T11:48:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1266311168 bytes 2024-09-16T11:48:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1315406848 bytes 2024-09-16T11:48:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1335719936 bytes 2024-09-16T11:48:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1368954880 bytes 2024-09-16T11:48:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1410634752 bytes 2024-09-16T11:48:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1461029888 bytes 2024-09-16T11:48:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1524086784 bytes 2024-09-16T11:48:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1556540416 bytes 2024-09-16T11:48:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:48:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:48:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:48:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-16T11:48:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_4 2024-09-16T11:48:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T11:48:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=1509652c2147 2024-09-16T11:48:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:48:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:48:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_4 present 1509652c2147 2024-09-16T11:48:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:48:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=1509652c2147 2024-09-16T11:48:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_4 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T11:48:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=1509652c2147 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1509652c2147 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 1509652c2147 -o 1509652c2147 = 1509652c2147 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-16T11:48:13.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:48:13.771+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d05403000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2024-09-16T11:48:13.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:48:13.771+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d05403000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2024-09-16T11:48:17.228 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.743672660207708) > weight (0.25) 2024-09-16T11:48:17.229 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7107357426634399) > weight (0.25) 2024-09-16T11:48:17.229 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4085298747662228) > weight (0.25) 2024-09-16T11:48:17.229 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.829333899676059) > weight (0.25) 2024-09-16T11:48:17.229 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-16T11:48:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:48:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:48:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T11:48:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_4 2024-09-16T11:48:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T11:48:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T11:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:48:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:48:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T11:48:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_4 2024-09-16T11:48:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T11:48:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T11:48:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:48:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-16T11:48:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-16T11:48:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:48:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:48:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T11:48:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_4 2024-09-16T11:48:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T11:48:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T11:48:35.115 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7679922054888837) > weight (0.25) 2024-09-16T11:48:35.115 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8815080469215777) > weight (0.25) 2024-09-16T11:48:35.115 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T11:48:35.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T11:48:35.116 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T11:48:35.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:48:35.116+0000 7fafcc5f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 67819) UID: 1000 2024-09-16T11:48:35.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:48:39.117 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T11:48:39.118 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:48:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:48:39.118 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T11:48:39.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T11:48:39.118 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T11:48:39.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T11:48:39.121 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-09-16T11:48:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_5 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:48:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:48:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:48:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:48:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2024-09-16T11:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-16T11:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-16T11:48:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:48:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-16T11:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_5': 2024-09-16T11:48:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:48:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:48:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:48:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 19a2ec32d0a2 2024-09-16T11:48:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.19a2ec32d0a2 2024-09-16T11:48:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:48:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:48:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:48:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:48:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:48:43 2024 2024-09-16T11:48:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:48:43 2024 2024-09-16T11:48:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:48:43 2024 2024-09-16T11:48:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:48:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:48:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 6716b49d-eaa9-4fca-919f-176309bb0077 2024-09-16T11:48:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:48:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-16T11:48:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:48:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=6 2024-09-16T11:48:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:48:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:48:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 43680768 bytes 2024-09-16T11:48:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 130795520 bytes 2024-09-16T11:48:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 190362624 bytes 2024-09-16T11:48:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 223917056 bytes 2024-09-16T11:48:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 224531456 bytes 2024-09-16T11:48:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 246164480 bytes 2024-09-16T11:48:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 271809536 bytes 2024-09-16T11:48:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 287560704 bytes 2024-09-16T11:48:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 301247488 bytes 2024-09-16T11:48:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 332272640 bytes 2024-09-16T11:48:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 402257920 bytes 2024-09-16T11:48:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 468771840 bytes 2024-09-16T11:48:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 495585280 bytes 2024-09-16T11:48:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 537552896 bytes 2024-09-16T11:48:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 585793536 bytes 2024-09-16T11:48:46.270 INFO:tasks.ceph.cluster2.osd.2.smithi103.stderr:2024-09-16T11:48:46.269+0000 7f1e20f72640 -1 reset not still connected to 0x55d52a725e10 2024-09-16T11:48:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 599246848 bytes 2024-09-16T11:48:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 631436288 bytes 2024-09-16T11:48:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 664435712 bytes 2024-09-16T11:48:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 741545984 bytes 2024-09-16T11:48:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 758992896 bytes 2024-09-16T11:48:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 789010432 bytes 2024-09-16T11:48:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 854400000 bytes 2024-09-16T11:48:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 883793920 bytes 2024-09-16T11:48:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 910470144 bytes 2024-09-16T11:48:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 933204992 bytes 2024-09-16T11:48:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 991744000 bytes 2024-09-16T11:48:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1006440448 bytes 2024-09-16T11:48:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1047923712 bytes 2024-09-16T11:48:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1096978432 bytes 2024-09-16T11:48:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1142601728 bytes 2024-09-16T11:48:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1172882432 bytes 2024-09-16T11:48:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1202395136 bytes 2024-09-16T11:48:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1229192192 bytes 2024-09-16T11:48:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1269752832 bytes 2024-09-16T11:48:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1331402752 bytes 2024-09-16T11:48:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1369489408 bytes 2024-09-16T11:48:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1383208960 bytes 2024-09-16T11:48:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1406055424 bytes 2024-09-16T11:48:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:48:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:48:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:48:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-16T11:48:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_5 2024-09-16T11:48:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T11:48:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=15099b461456 2024-09-16T11:48:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:48:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_5 present 15099b461456 2024-09-16T11:48:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=15099b461456 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_5 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T11:48:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=15099b461456 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 15099b461456 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 15099b461456 -o 15099b461456 = 15099b461456 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-16T11:48:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-16T11:48:50.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:48:50.114+0000 7f097a5eb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99cb040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:48:50.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:48:50.114+0000 7f0980df8640 -1 rbd::mirror::ImageReplayer: 0x5591d952e000 [3/6716b49d-eaa9-4fca-919f-176309bb0077] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:48:50.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:48:50.844+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d05649000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2024-09-16T11:48:50.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:48:50.844+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d05649000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2024-09-16T11:48:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:48:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-16T11:48:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:48:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_5 2024-09-16T11:48:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T11:48:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T11:48:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:48:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:48:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:48:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_5 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T11:48:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:48:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T11:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-16T11:48:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-16T11:48:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-16T11:48:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:48:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:48:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T11:48:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:48:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:48:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_5 2024-09-16T11:48:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T11:48:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T11:49:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:49:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:49:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:49:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_6 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2024-09-16T11:49:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-16T11:49:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:49:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-16T11:49:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:49:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-16T11:49:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_6': 2024-09-16T11:49:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:49:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:49:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:49:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 19ba6033984 2024-09-16T11:49:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.19ba6033984 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:49:20 2024 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:49:20 2024 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:49:20 2024 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f56cf663-fda2-4ee1-b4f7-90edac2fb26a 2024-09-16T11:49:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:49:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-16T11:49:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:49:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=5 2024-09-16T11:49:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:49:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:49:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 73703424 bytes 2024-09-16T11:49:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 100892672 bytes 2024-09-16T11:49:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 171421696 bytes 2024-09-16T11:49:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 194612224 bytes 2024-09-16T11:49:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 238385152 bytes 2024-09-16T11:49:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 349350912 bytes 2024-09-16T11:49:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 392394752 bytes 2024-09-16T11:49:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 410338304 bytes 2024-09-16T11:49:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 447338496 bytes 2024-09-16T11:49:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 454128640 bytes 2024-09-16T11:49:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 483536896 bytes 2024-09-16T11:49:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 509656064 bytes 2024-09-16T11:49:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 615354368 bytes 2024-09-16T11:49:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 695139328 bytes 2024-09-16T11:49:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 802649088 bytes 2024-09-16T11:49:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 821337088 bytes 2024-09-16T11:49:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 852027392 bytes 2024-09-16T11:49:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 920659968 bytes 2024-09-16T11:49:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 943314944 bytes 2024-09-16T11:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 971803648 bytes 2024-09-16T11:49:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1005100032 bytes 2024-09-16T11:49:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1073013760 bytes 2024-09-16T11:49:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1096014848 bytes 2024-09-16T11:49:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1143455744 bytes 2024-09-16T11:49:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1198358528 bytes 2024-09-16T11:49:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1224093696 bytes 2024-09-16T11:49:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1257519104 bytes 2024-09-16T11:49:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1276276736 bytes 2024-09-16T11:49:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1314492416 bytes 2024-09-16T11:49:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:49:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:49:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:49:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:49:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_6 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T11:49:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:49:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=1509d9194264 2024-09-16T11:49:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:49:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:49:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_6 present 1509d9194264 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=1509d9194264 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:49:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:49:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T11:49:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:49:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:49:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_6 2024-09-16T11:49:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T11:49:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=1509d9194264 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1509d9194264 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 1509d9194264 -o 1509d9194264 = 1509d9194264 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-16T11:49:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:49:26.731+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d050be400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2024-09-16T11:49:26.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:49:26.731+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d050be400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2024-09-16T11:49:31.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:49:31.248+0000 7f0979dea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5591d99a9380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T11:49:31.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:49:31.248+0000 7f0980df8640 -1 rbd::mirror::ImageReplayer: 0x5591d952e000 [3/f56cf663-fda2-4ee1-b4f7-90edac2fb26a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T11:49:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:49:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-16T11:49:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:49:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_6 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T11:49:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:49:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T11:49:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:49:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:49:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:49:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-16T11:49:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_6 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T11:49:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:49:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T11:49:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:49:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:49:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:49:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:49:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-16T11:49:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-16T11:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-16T11:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:49:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:49:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T11:49:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_6 2024-09-16T11:49:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T11:49:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T11:49:32.230 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T11:49:32.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T11:49:32.231 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T11:49:32.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:49:32.230+0000 7f9c337f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 64693) UID: 1000 2024-09-16T11:49:32.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:49:37.231 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T11:49:37.231 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:49:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:49:37.232 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T11:49:37.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T11:49:37.232 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T11:49:37.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T11:49:37.236 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-16T11:49:43.122 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8224963479720411) > weight (0.25) 2024-09-16T11:49:43.122 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.401990874284443) > weight (0.25) 2024-09-16T11:49:43.122 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.34738710694655384) > weight (0.25) 2024-09-16T11:49:43.122 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T11:49:43.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T11:49:43.123 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T11:49:43.123 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T11:49:43.123 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:49:43.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:49:43.122+0000 7f09815f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 55018) UID: 1000 2024-09-16T11:49:43.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:49:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:49:49.225 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T11:49:49.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T11:49:49.225 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T11:49:49.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T11:49:49.228 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-16T11:49:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_7 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:49:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2024-09-16T11:49:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-16T11:49:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:49:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:49:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-16T11:49:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:49:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_7': 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 19e22e1d7fcb 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.19e22e1d7fcb 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:49:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:49:55 2024 2024-09-16T11:49:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:49:55 2024 2024-09-16T11:49:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:49:55 2024 2024-09-16T11:49:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:49:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:49:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: f1619a22-1530-456e-9701-769e7980d25f 2024-09-16T11:49:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:49:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-16T11:49:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:49:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:49:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:49:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:49:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:49:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:49:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:49:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 57713664 bytes 2024-09-16T11:49:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 107105280 bytes 2024-09-16T11:49:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 162465792 bytes 2024-09-16T11:49:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 251445248 bytes 2024-09-16T11:49:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 267553792 bytes 2024-09-16T11:49:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 293665792 bytes 2024-09-16T11:49:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 385671168 bytes 2024-09-16T11:49:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 401426432 bytes 2024-09-16T11:49:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 425440256 bytes 2024-09-16T11:49:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 442502144 bytes 2024-09-16T11:49:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 525228032 bytes 2024-09-16T11:49:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 559556608 bytes 2024-09-16T11:49:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 579036160 bytes 2024-09-16T11:49:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 625410048 bytes 2024-09-16T11:49:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 692219904 bytes 2024-09-16T11:49:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 701860864 bytes 2024-09-16T11:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 761041920 bytes 2024-09-16T11:49:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 801838080 bytes 2024-09-16T11:49:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 839277568 bytes 2024-09-16T11:49:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 910438400 bytes 2024-09-16T11:49:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 941151232 bytes 2024-09-16T11:49:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1019370496 bytes 2024-09-16T11:49:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1089480704 bytes 2024-09-16T11:49:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1096383488 bytes 2024-09-16T11:50:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1136867328 bytes 2024-09-16T11:50:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1197908992 bytes 2024-09-16T11:50:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1217215488 bytes 2024-09-16T11:50:00.616 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1245830144 bytes 2024-09-16T11:50:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1285286912 bytes 2024-09-16T11:50:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1351138304 bytes 2024-09-16T11:50:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1407848448 bytes 2024-09-16T11:50:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1430063104 bytes 2024-09-16T11:50:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1492166656 bytes 2024-09-16T11:50:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1526867968 bytes 2024-09-16T11:50:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1549547520 bytes 2024-09-16T11:50:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1614224384 bytes 2024-09-16T11:50:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1685255168 bytes 2024-09-16T11:50:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1703734272 bytes 2024-09-16T11:50:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1738240000 bytes 2024-09-16T11:50:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1767017472 bytes 2024-09-16T11:50:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1804080128 bytes 2024-09-16T11:50:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1810620416 bytes 2024-09-16T11:50:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1844817920 bytes 2024-09-16T11:50:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1896922112 bytes 2024-09-16T11:50:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2002232320 bytes 2024-09-16T11:50:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2029530112 bytes 2024-09-16T11:50:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2062527488 bytes 2024-09-16T11:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2067343360 bytes 2024-09-16T11:50:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2123722752 bytes 2024-09-16T11:50:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2157918208 bytes 2024-09-16T11:50:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2183012352 bytes 2024-09-16T11:50:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2243860480 bytes 2024-09-16T11:50:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:50:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:50:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T11:50:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_7 2024-09-16T11:50:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T11:50:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=150970833822 2024-09-16T11:50:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:50:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_7 present 150970833822 2024-09-16T11:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=150970833822 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_7 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T11:50:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=150970833822 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 150970833822 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 150970833822 -o 150970833822 = 150970833822 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-16T11:50:05.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:50:05.820+0000 7fe7c9fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d03f6f800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2024-09-16T11:50:05.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:50:05.820+0000 7fe7c9fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d03f6f800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2024-09-16T11:50:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:50:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-16T11:50:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_7 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T11:50:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:10.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:50:10.849+0000 7fe8e51f8640 -1 rbd::mirror::ImageMap: 0x55dbf3ad2480 handle_peer_ack_remove: failed to remove global_image_id=f1619a22-1530-456e-9701-769e7980d25f 2024-09-16T11:50:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T11:50:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:50:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:50:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:50:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:50:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T11:50:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_7 2024-09-16T11:50:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T11:50:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T11:50:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:50:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:50:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:50:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-16T11:50:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-16T11:50:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-16T11:50:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T11:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_7 2024-09-16T11:50:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T11:50:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T11:50:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_8 2024-09-16T11:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-16T11:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2024-09-16T11:50:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-16T11:50:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:50:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-16T11:50:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:50:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-16T11:50:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_8': 2024-09-16T11:50:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:50:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:50:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:50:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 19faf806ec73 2024-09-16T11:50:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.19faf806ec73 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:50:32 2024 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:50:32 2024 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:50:32 2024 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 08529107-06c7-4016-88ab-71aca78eadeb 2024-09-16T11:50:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:50:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-16T11:50:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:50:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=13 2024-09-16T11:50:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:50:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:50:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 833536 bytes 2024-09-16T11:50:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 50630656 bytes 2024-09-16T11:50:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 65452032 bytes 2024-09-16T11:50:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 66043904 bytes 2024-09-16T11:50:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 91678720 bytes 2024-09-16T11:50:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 135084032 bytes 2024-09-16T11:50:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 173123584 bytes 2024-09-16T11:50:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 215581696 bytes 2024-09-16T11:50:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 246070272 bytes 2024-09-16T11:50:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 320423936 bytes 2024-09-16T11:50:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 387627008 bytes 2024-09-16T11:50:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 432700416 bytes 2024-09-16T11:50:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 480267264 bytes 2024-09-16T11:50:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 535534592 bytes 2024-09-16T11:50:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 596705280 bytes 2024-09-16T11:50:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 627828736 bytes 2024-09-16T11:50:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 659599360 bytes 2024-09-16T11:50:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 671180800 bytes 2024-09-16T11:50:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 721635328 bytes 2024-09-16T11:50:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 800680960 bytes 2024-09-16T11:50:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 845978624 bytes 2024-09-16T11:50:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 893622272 bytes 2024-09-16T11:50:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 918329344 bytes 2024-09-16T11:50:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1036673024 bytes 2024-09-16T11:50:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1051897856 bytes 2024-09-16T11:50:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1061991424 bytes 2024-09-16T11:50:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1073861632 bytes 2024-09-16T11:50:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1115667456 bytes 2024-09-16T11:50:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1166192640 bytes 2024-09-16T11:50:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1216443392 bytes 2024-09-16T11:50:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1218304000 bytes 2024-09-16T11:50:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1272076288 bytes 2024-09-16T11:50:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1372991488 bytes 2024-09-16T11:50:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1411809280 bytes 2024-09-16T11:50:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1451081728 bytes 2024-09-16T11:50:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1459798016 bytes 2024-09-16T11:50:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1461976064 bytes 2024-09-16T11:50:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1506731008 bytes 2024-09-16T11:50:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1513135104 bytes 2024-09-16T11:50:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1613229056 bytes 2024-09-16T11:50:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 1640838144 bytes 2024-09-16T11:50:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 1654748160 bytes 2024-09-16T11:50:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 1703382016 bytes 2024-09-16T11:50:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 1713298432 bytes 2024-09-16T11:50:40.860 INFO:tasks.ceph.cluster2.osd.0.smithi103.stderr:2024-09-16T11:50:40.858+0000 7f48ae988640 -1 reset not still connected to 0x5559cc8a29c0 2024-09-16T11:50:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 1732801536 bytes 2024-09-16T11:50:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 1756212224 bytes 2024-09-16T11:50:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 1775902720 bytes 2024-09-16T11:50:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 1804039168 bytes 2024-09-16T11:50:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 1841457152 bytes 2024-09-16T11:50:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 1928859648 bytes 2024-09-16T11:50:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 1942157312 bytes 2024-09-16T11:50:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 1960307712 bytes 2024-09-16T11:50:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2033187840 bytes 2024-09-16T11:50:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2050934784 bytes 2024-09-16T11:50:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2094387200 bytes 2024-09-16T11:50:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2178649088 bytes 2024-09-16T11:50:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2231576576 bytes 2024-09-16T11:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2314816512 bytes 2024-09-16T11:50:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2350684160 bytes 2024-09-16T11:50:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2376306688 bytes 2024-09-16T11:50:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2461237248 bytes 2024-09-16T11:50:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2493104128 bytes 2024-09-16T11:50:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2540119040 bytes 2024-09-16T11:50:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2587380736 bytes 2024-09-16T11:50:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2651852800 bytes 2024-09-16T11:50:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2751295488 bytes 2024-09-16T11:50:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6700 IOs, 2751917056 bytes 2024-09-16T11:50:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6800 IOs, 2777026560 bytes 2024-09-16T11:50:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6900 IOs, 2819188736 bytes 2024-09-16T11:50:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7000 IOs, 2832315392 bytes 2024-09-16T11:50:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7100 IOs, 2832916480 bytes 2024-09-16T11:50:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7200 IOs, 2870060032 bytes 2024-09-16T11:50:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7300 IOs, 2889333760 bytes 2024-09-16T11:50:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7400 IOs, 2945985536 bytes 2024-09-16T11:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7500 IOs, 3007131648 bytes 2024-09-16T11:50:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7600 IOs, 3054978048 bytes 2024-09-16T11:50:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7700 IOs, 3113114624 bytes 2024-09-16T11:50:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7800 IOs, 3143126016 bytes 2024-09-16T11:50:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:7900 IOs, 3244115968 bytes 2024-09-16T11:50:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:50:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:50:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:50:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-16T11:50:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_8 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T11:50:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=15092cf2d181 2024-09-16T11:50:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:50:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:50:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_8 present 15092cf2d181 2024-09-16T11:50:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:50:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=15092cf2d181 2024-09-16T11:50:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:50:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:50:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T11:50:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_8 2024-09-16T11:50:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T11:50:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=15092cf2d181 2024-09-16T11:50:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 15092cf2d181 2024-09-16T11:50:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 15092cf2d181 -o 15092cf2d181 = 15092cf2d181 2024-09-16T11:50:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:50:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:50:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-16T11:50:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-16T11:50:47.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:50:47.795+0000 7fe7c9fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d056cf400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2024-09-16T11:50:47.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:50:47.795+0000 7fe7c9fde640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d056cf400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2024-09-16T11:50:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:50:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-16T11:50:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:50:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_8 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T11:50:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T11:50:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:50:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:50:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:50:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-16T11:50:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:50:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_8 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T11:50:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:50:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T11:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-16T11:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-16T11:50:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-16T11:50:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:50:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:50:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T11:50:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:50:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:50:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_8 2024-09-16T11:50:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T11:50:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T11:51:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_9 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:51:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:51:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:51:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2024-09-16T11:51:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-16T11:51:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:51:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-16T11:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:51:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_9': 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1a1262df7698 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1a1262df7698 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:51:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:51:16 2024 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:51:16 2024 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:51:16 2024 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 7be69e2e-c070-4c38-a39a-870853ad2ace 2024-09-16T11:51:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:51:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-16T11:51:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:51:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=9 2024-09-16T11:51:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:51:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:51:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 55555072 bytes 2024-09-16T11:51:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 57565184 bytes 2024-09-16T11:51:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 102377472 bytes 2024-09-16T11:51:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 131467264 bytes 2024-09-16T11:51:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 155746304 bytes 2024-09-16T11:51:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 159102976 bytes 2024-09-16T11:51:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 230756352 bytes 2024-09-16T11:51:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 302080000 bytes 2024-09-16T11:51:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 319821824 bytes 2024-09-16T11:51:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 408355840 bytes 2024-09-16T11:51:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 511304704 bytes 2024-09-16T11:51:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 536586240 bytes 2024-09-16T11:51:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 578073600 bytes 2024-09-16T11:51:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 650576896 bytes 2024-09-16T11:51:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 736346112 bytes 2024-09-16T11:51:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 817321984 bytes 2024-09-16T11:51:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 819412992 bytes 2024-09-16T11:51:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 933569536 bytes 2024-09-16T11:51:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 971122688 bytes 2024-09-16T11:51:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1007134720 bytes 2024-09-16T11:51:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1028137984 bytes 2024-09-16T11:51:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1058876416 bytes 2024-09-16T11:51:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1135102976 bytes 2024-09-16T11:51:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1154875392 bytes 2024-09-16T11:51:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1203407872 bytes 2024-09-16T11:51:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1270156288 bytes 2024-09-16T11:51:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1322426368 bytes 2024-09-16T11:51:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1323049984 bytes 2024-09-16T11:51:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1398594560 bytes 2024-09-16T11:51:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1435524096 bytes 2024-09-16T11:51:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1478468608 bytes 2024-09-16T11:51:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1520450560 bytes 2024-09-16T11:51:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1588258816 bytes 2024-09-16T11:51:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1628466176 bytes 2024-09-16T11:51:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1680661504 bytes 2024-09-16T11:51:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1705784320 bytes 2024-09-16T11:51:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1757330432 bytes 2024-09-16T11:51:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1788954624 bytes 2024-09-16T11:51:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1836672000 bytes 2024-09-16T11:51:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1922255872 bytes 2024-09-16T11:51:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 2011613184 bytes 2024-09-16T11:51:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 2045740032 bytes 2024-09-16T11:51:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2106647552 bytes 2024-09-16T11:51:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2165495808 bytes 2024-09-16T11:51:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2176568320 bytes 2024-09-16T11:51:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2282252288 bytes 2024-09-16T11:51:26.237 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8780822792110008) > weight (0.25) 2024-09-16T11:51:26.237 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T11:51:26.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T11:51:26.237 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T11:51:26.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:51:26.235+0000 7fa4a63f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 67465) UID: 1000 2024-09-16T11:51:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2349916160 bytes 2024-09-16T11:51:26.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:51:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2359881728 bytes 2024-09-16T11:51:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2404588544 bytes 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:51:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:51:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T11:51:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:51:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:51:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_9 2024-09-16T11:51:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T11:51:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:51:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=15097e1c9516 2024-09-16T11:51:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:51:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:51:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_9 present 15097e1c9516 2024-09-16T11:51:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:51:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=15097e1c9516 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_9 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T11:51:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:51:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=15097e1c9516 2024-09-16T11:51:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 15097e1c9516 2024-09-16T11:51:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 15097e1c9516 -o 15097e1c9516 = 15097e1c9516 2024-09-16T11:51:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:51:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:51:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:51:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-16T11:51:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-16T11:51:27.238 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T11:51:27.238 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:51:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:51:27.239 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T11:51:27.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T11:51:27.239 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T11:51:27.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T11:51:27.242 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-16T11:51:27.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:51:27.822+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d0502e400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2024-09-16T11:51:27.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:51:27.822+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d0502e400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=388, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2024-09-16T11:51:28.229 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T11:51:28.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T11:51:28.229 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T11:51:28.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:51:28.228+0000 7fe8e59f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 64456) UID: 1000 2024-09-16T11:51:28.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:51:31.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:51:31.246+0000 7fca6cbf7640 -1 rbd::mirror::ImageMap: 0x55cd2bf107e0 handle_peer_ack_remove: failed to remove global_image_id=7be69e2e-c070-4c38-a39a-870853ad2ace 2024-09-16T11:51:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:51:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:51:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_9 2024-09-16T11:51:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T11:51:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:51:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T11:51:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:51:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:51:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:51:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:51:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_9 2024-09-16T11:51:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T11:51:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:51:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T11:51:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-16T11:51:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-16T11:51:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:51:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T11:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_9 2024-09-16T11:51:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T11:51:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T11:51:36.230 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T11:51:36.230 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T11:51:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:51:36.230 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T11:51:36.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T11:51:36.231 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T11:51:36.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T11:51:36.234 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-16T11:51:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for i in `seq 1 10` 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image=image_10 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local size=128 2024-09-16T11:51:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -n 512M ']' 2024-09-16T11:51:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ size=512M 2024-09-16T11:51:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ shift 2024-09-16T11:51:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2024-09-16T11:51:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-16T11:51:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:51:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local mode=snapshot 2024-09-16T11:51:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-16T11:51:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring enabled 2024-09-16T11:51:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:rbd image 'image_10': 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: size 512 MiB in 128 objects 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: order 22 (4 MiB objects) 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: snapshot_count: 1 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: id: 1a364ca63ffa 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: block_name_prefix: rbd_data.1a364ca63ffa 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: format: 2 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: features: layering, exclusive-lock 2024-09-16T11:51:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: op_features: 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: flags: 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: create_timestamp: Mon Sep 16 11:51:57 2024 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: access_timestamp: Mon Sep 16 11:51:57 2024 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: modify_timestamp: Mon Sep 16 11:51:57 2024 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring state: enabled 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring mode: snapshot 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring global id: 14a640f7-c7c4-421a-ae0f-94ea6a4dc5da 2024-09-16T11:51:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout: mirroring primary: true 2024-09-16T11:51:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-16T11:51:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:51:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:51:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:51:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T11:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local duration=11 2024-09-16T11:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-16T11:51:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:100 IOs, 19853312 bytes 2024-09-16T11:51:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:200 IOs, 79130624 bytes 2024-09-16T11:51:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:300 IOs, 197648384 bytes 2024-09-16T11:51:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:400 IOs, 275440640 bytes 2024-09-16T11:51:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:500 IOs, 309793792 bytes 2024-09-16T11:51:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:600 IOs, 369884160 bytes 2024-09-16T11:51:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:700 IOs, 382326784 bytes 2024-09-16T11:51:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:800 IOs, 462298112 bytes 2024-09-16T11:51:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:900 IOs, 556257280 bytes 2024-09-16T11:51:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1000 IOs, 594791424 bytes 2024-09-16T11:52:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1100 IOs, 633182208 bytes 2024-09-16T11:52:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1200 IOs, 708002816 bytes 2024-09-16T11:52:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1300 IOs, 743635968 bytes 2024-09-16T11:52:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1400 IOs, 852535296 bytes 2024-09-16T11:52:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1500 IOs, 879319040 bytes 2024-09-16T11:52:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1600 IOs, 901365760 bytes 2024-09-16T11:52:01.177 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1700 IOs, 987236352 bytes 2024-09-16T11:52:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1800 IOs, 1026772992 bytes 2024-09-16T11:52:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:1900 IOs, 1042823168 bytes 2024-09-16T11:52:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2000 IOs, 1118188544 bytes 2024-09-16T11:52:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2100 IOs, 1126435840 bytes 2024-09-16T11:52:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2200 IOs, 1174591488 bytes 2024-09-16T11:52:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2300 IOs, 1201517568 bytes 2024-09-16T11:52:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2400 IOs, 1240847360 bytes 2024-09-16T11:52:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2500 IOs, 1278862336 bytes 2024-09-16T11:52:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2600 IOs, 1330382848 bytes 2024-09-16T11:52:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2700 IOs, 1336352768 bytes 2024-09-16T11:52:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2800 IOs, 1386727424 bytes 2024-09-16T11:52:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:2900 IOs, 1402072064 bytes 2024-09-16T11:52:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3000 IOs, 1407294464 bytes 2024-09-16T11:52:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3100 IOs, 1440631808 bytes 2024-09-16T11:52:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3200 IOs, 1519448064 bytes 2024-09-16T11:52:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3300 IOs, 1550321664 bytes 2024-09-16T11:52:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3400 IOs, 1652162560 bytes 2024-09-16T11:52:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3500 IOs, 1665001472 bytes 2024-09-16T11:52:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3600 IOs, 1704123392 bytes 2024-09-16T11:52:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3700 IOs, 1755927552 bytes 2024-09-16T11:52:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3800 IOs, 1808335872 bytes 2024-09-16T11:52:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:3900 IOs, 1902481408 bytes 2024-09-16T11:52:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4000 IOs, 1957986304 bytes 2024-09-16T11:52:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4100 IOs, 2012580864 bytes 2024-09-16T11:52:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4200 IOs, 2067556352 bytes 2024-09-16T11:52:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4300 IOs, 2074320896 bytes 2024-09-16T11:52:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4400 IOs, 2104517632 bytes 2024-09-16T11:52:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4500 IOs, 2134811648 bytes 2024-09-16T11:52:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4600 IOs, 2160772096 bytes 2024-09-16T11:52:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4700 IOs, 2198704128 bytes 2024-09-16T11:52:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4800 IOs, 2244536320 bytes 2024-09-16T11:52:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:4900 IOs, 2305038336 bytes 2024-09-16T11:52:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5000 IOs, 2329924608 bytes 2024-09-16T11:52:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5100 IOs, 2387477504 bytes 2024-09-16T11:52:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5200 IOs, 2469862400 bytes 2024-09-16T11:52:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5300 IOs, 2473675776 bytes 2024-09-16T11:52:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5400 IOs, 2483367936 bytes 2024-09-16T11:52:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5500 IOs, 2494858240 bytes 2024-09-16T11:52:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5600 IOs, 2505385984 bytes 2024-09-16T11:52:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5700 IOs, 2550159360 bytes 2024-09-16T11:52:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5800 IOs, 2550750208 bytes 2024-09-16T11:52:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:5900 IOs, 2586588160 bytes 2024-09-16T11:52:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6000 IOs, 2621663232 bytes 2024-09-16T11:52:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6100 IOs, 2654739456 bytes 2024-09-16T11:52:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6200 IOs, 2704191488 bytes 2024-09-16T11:52:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6300 IOs, 2732546048 bytes 2024-09-16T11:52:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6400 IOs, 2774765568 bytes 2024-09-16T11:52:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6500 IOs, 2841744384 bytes 2024-09-16T11:52:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:6600 IOs, 2849674240 bytes 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=124 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=present 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_10 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T11:52:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:52:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id=15093112455 2024-09-16T11:52:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:52:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:52:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_10 present 15093112455 2024-09-16T11:52:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=present 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id=15093112455 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_10 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T11:52:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:52:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id=15093112455 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 15093112455 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -z 15093112455 -o 15093112455 = 15093112455 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_state=present 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test present = present 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-16T11:52:09.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:09.260+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d050f1c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2024-09-16T11:52:09.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:09.260+0000 7fe7c97dd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x555d050f1c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=390, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=04869fab-2245-439f-841b-481bc7c8cad8, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2024-09-16T11:52:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:Mirroring disabled 2024-09-16T11:52:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-16T11:52:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local state=deleted 2024-09-16T11:52:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:52:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local s 2024-09-16T11:52:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:52:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T11:52:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_10 2024-09-16T11:52:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T11:52:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:52:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T11:52:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ image_id= 2024-09-16T11:52:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:52:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0.1 2024-09-16T11:52:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-16T11:52:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local test_state=deleted 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image_id= 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_state=deleted 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local current_image_id 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local image=image_10 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T11:52:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:52:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ current_image_id= 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n '' 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test deleted = deleted 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-16T11:52:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-16T11:52:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:52:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T11:52:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local image=image_10 2024-09-16T11:52:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T11:52:14.524 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T11:52:30.243 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.47979887521407405) > weight (0.25) 2024-09-16T11:52:30.244 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.899771856858117) > weight (0.25) 2024-09-16T11:52:30.244 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7528219028714849) > weight (0.25) 2024-09-16T11:52:30.244 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5537812083466198) > weight (0.25) 2024-09-16T11:52:30.244 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-16T11:52:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T11:52:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-16T11:52:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ tee -a /tmp/tmp.2dE0YG6s4q/rbd-mirror.test.log 2024-09-16T11:52:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ date '+%F %T' 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo 2024-09-16 11:52:38 TEST: check if removed 'images'\''' OMAP are removed 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:2024-09-16 11:52:38 TEST: check if removed images' OMAP are removed 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-16T11:52:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local filter=status_global 2024-09-16T11:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:52:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-16T11:52:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local filter=status_global 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:52:38.969 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -c status_global 2024-09-16T11:52:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:52:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:52:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local filter=image_ 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:52:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local filter=image_ 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:52:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -c image_ 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T11:52:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local filter=image_map 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local filter=image_map 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster1 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:52:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -c image_map 2024-09-16T11:52:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:52:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:52:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:52:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local filter=status_global 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:52:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster2 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local filter=status_global 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster2 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:52:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -c status_global 2024-09-16T11:52:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:52:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:52:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local filter=image_ 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster2 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local filter=image_ 2024-09-16T11:52:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T11:52:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster2 2024-09-16T11:52:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:52:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -c image_ 2024-09-16T11:52:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:52:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:52:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:52:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local pool=mirror 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local filter=image_map 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ sleep 0 2024-09-16T11:52:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-16T11:52:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster2 2024-09-16T11:52:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local filter=image_map 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local cluster=cluster2 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local pool=mirror 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:52:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ grep -c image_map 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test 0 = 0 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ error_code=0 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set -e 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 0 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ cleanup 0 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local error_code=0 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ set +e 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T11:52:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z '' ']' 2024-09-16T11:52:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster instance 2024-09-16T11:52:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:52:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T11:52:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror' does not exist 2024-09-16T11:52:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:52:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T11:52:40.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.322+0000 7f405d1f8640 -1 rbd::mirror::InstanceWatcher: 0x55eec83cb500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5499 object: (2) No such file or directory 2024-09-16T11:52:40.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.322+0000 7f405d1f8640 -1 rbd::mirror::InstanceWatcher: 0x55eec83cb880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5499 object: (2) No such file or directory 2024-09-16T11:52:40.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.323+0000 7f40561ea640 -1 rbd::mirror::InstanceWatcher: 0x55eec83cb500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:40.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.323+0000 7f40561ea640 -1 rbd::mirror::InstanceWatcher: 0x55eec83cb880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:40.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.326+0000 7f40569eb640 -1 rbd::mirror::MirrorStatusUpdater 0x55eecb55e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:40.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.326+0000 7f405c9f7640 -1 rbd::mirror::NamespaceReplayer: 0x55eecb55c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:40.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.326+0000 7f40569eb640 -1 rbd::mirror::MirrorStatusUpdater 0x55eecb55e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:40.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.326+0000 7f405d1f8640 -1 rbd::mirror::NamespaceReplayer: 0x55eecb55c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:40.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.702+0000 7f405c9f7640 -1 rbd::mirror::LeaderWatcher: 0x55eeca9c4600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:40.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.703+0000 7f405c9f7640 -1 rbd::mirror::InstanceWatcher: 0x55eec83cb180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5499 object: (2) No such file or directory 2024-09-16T11:52:40.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.703+0000 7f40569eb640 -1 rbd::mirror::InstanceWatcher: 0x55eec83cb180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:40.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.707+0000 7f40561ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55eec9e05440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:40.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:40.707+0000 7f405d1f8640 -1 rbd::mirror::NamespaceReplayer: 0x55eeca9ae4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror' does not exist 2024-09-16T11:52:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:52:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T11:52:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror_parent' does not exist 2024-09-16T11:52:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ CEPH_ARGS= 2024-09-16T11:52:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T11:52:42.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:42.308+0000 7fe7d67f7640 -1 rbd::mirror::LeaderWatcher: 0x555d04115800 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T11:52:42.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:42.308+0000 7fe7c2fd0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555d052c1d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:42.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:42.308+0000 7fe7c27cf640 -1 rbd::mirror::PoolWatcher: 0x555d057283c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:42.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:42.311+0000 7fe7d67f7640 -1 rbd::mirror::LeaderWatcher: 0x555d04115800 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T11:52:42.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:42.311+0000 7fe7d67f7640 -1 rbd::mirror::LeaderWatcher: 0x555d04115800 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T11:52:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:pool 'mirror_parent' does not exist 2024-09-16T11:52:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T11:52:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirrors cluster1 2024-09-16T11:52:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1 2024-09-16T11:52:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 0 3 2024-09-16T11:52:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T11:52:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:0 2024-09-16T11:52:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T11:52:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:1 2024-09-16T11:52:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster1:2 '' 2024-09-16T11:52:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:2 2024-09-16T11:52:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster1:3 '' 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster1:3 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T11:52:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirrors cluster2 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:++ seq 0 3 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster2:0 '' 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:0 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster2:1 '' 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:1 2024-09-16T11:52:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster2:2 '' 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:2 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ stop_mirror cluster2:3 '' 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local cluster=cluster2:3 2024-09-16T11:52:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ local sig= 2024-09-16T11:52:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test -n 1 2024-09-16T11:52:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ return 2024-09-16T11:52:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' -z 1 ']' 2024-09-16T11:52:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ test '' = /tmp/tmp.2dE0YG6s4q 2024-09-16T11:52:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ rm -Rf /tmp/tmp.2dE0YG6s4q 2024-09-16T11:52:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stdout:OK 2024-09-16T11:52:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:52:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ echo OK 2024-09-16T11:52:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi103.stderr:+ exit 0 2024-09-16T11:52:42.878 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:52:42.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-16T11:52:42.945 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-16T11:52:42.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-16T11:52:43.251 DEBUG:teuthology.parallel:result is None 2024-09-16T11:52:43.252 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T11:52:43.275 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T11:52:43.275 DEBUG:teuthology.orchestra.run.smithi103:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:52:43.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.279+0000 7fca6cbf7640 -1 rbd::mirror::LeaderWatcher: 0x55cd2b3e3800 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T11:52:43.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.279+0000 7fca593d0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cd2a784480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.279+0000 7fca593d0640 -1 rbd::mirror::PoolWatcher: 0x55cd281e01e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.280+0000 7fca52bc3640 -1 rbd::mirror::PoolWatcher: 0x55cd2c140000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.280+0000 7fca6cbf7640 -1 rbd::mirror::LeaderWatcher: 0x55cd2b3e3800 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T11:52:43.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.280+0000 7fca6cbf7640 -1 rbd::mirror::LeaderWatcher: 0x55cd2b3e3800 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T11:52:43.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca6d3f8640 -1 rbd::mirror::LeaderWatcher: 0x55cd2a6d0600 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T11:52:43.348 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T11:52:43.348 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-16T11:52:43.359 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-16T11:52:43.359 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-09-16T11:52:43.359 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-16T11:52:43.360 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-16T11:52:43.369 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-16T11:52:43.369 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2024-09-16T11:52:43.369 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-16T11:52:43.369 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.379 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-16T11:52:43.379 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca66beb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cd2c142240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca66beb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cd2a7858c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca66beb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55cd2a7846c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca663ea640 -1 rbd::mirror::PoolWatcher: 0x55cd2c1403c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca663ea640 -1 rbd::mirror::PoolWatcher: 0x55cd2c140780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.345+0000 7fca663ea640 -1 rbd::mirror::PoolWatcher: 0x55cd2c1405a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.346+0000 7fca5fbdd640 -1 rbd::mirror::PoolWatcher: 0x55cd2c140b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.346+0000 7fca5fbdd640 -1 rbd::mirror::PoolWatcher: 0x55cd2c140f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.346+0000 7fca603de640 -1 rbd::mirror::PoolWatcher: 0x55cd2c140960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.346+0000 7fca6d3f8640 -1 rbd::mirror::LeaderWatcher: 0x55cd2a6d0600 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T11:52:43.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.346+0000 7fca6d3f8640 -1 rbd::mirror::LeaderWatcher: 0x55cd2a6d0600 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.378+0000 7f91c7bf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 73024) UID: 1000 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.378+0000 7f91c73f8640 -1 rbd::mirror::LeaderWatcher: 0x563b4636f800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.378+0000 7f91c73f8640 -1 rbd::mirror::InstanceWatcher: 0x563b43045880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6620 object: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.378+0000 7f91c6bf7640 -1 rbd::mirror::InstanceWatcher: 0x563b43045500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6620 object: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91c73f8640 -1 rbd::mirror::InstanceWatcher: 0x563b43045c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6623 object: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91c0beb640 -1 rbd::mirror::InstanceWatcher: 0x563b43045880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91c0beb640 -1 rbd::mirror::InstanceWatcher: 0x563b43045500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91b33d0640 -1 rbd::mirror::InstanceWatcher: 0x563b43045c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91b9bdd640 -1 rbd::mirror::MirrorStatusUpdater 0x563b4309bc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91b9bdd640 -1 rbd::mirror::MirrorStatusUpdater 0x563b461c6360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91c73f8640 -1 rbd::mirror::NamespaceReplayer: 0x563b461c2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91c73f8640 -1 rbd::mirror::NamespaceReplayer: 0x563b461c21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91ac3c2640 -1 rbd::mirror::MirrorStatusUpdater 0x563b4309be60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.379+0000 7f91c6bf7640 -1 rbd::mirror::NamespaceReplayer: 0x563b46e9e4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.380+0000 7f91c03ea640 -1 rbd::mirror::MirrorStatusUpdater 0x563b461c6000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.380+0000 7f91c03ea640 -1 rbd::mirror::MirrorStatusUpdater 0x563b461c6120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.380+0000 7f91c73f8640 -1 rbd::mirror::NamespaceReplayer: 0x563b461c2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.380+0000 7f91c73f8640 -1 rbd::mirror::NamespaceReplayer: 0x563b461c21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.380+0000 7f91b2bcf640 -1 rbd::mirror::MirrorStatusUpdater 0x563b46326240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.380+0000 7f91c6bf7640 -1 rbd::mirror::NamespaceReplayer: 0x563b46e9e4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.381+0000 7f91c73f8640 -1 rbd::mirror::LeaderWatcher: 0x563b45622600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.382+0000 7f91c73f8640 -1 rbd::mirror::InstanceWatcher: 0x563b43045180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6620 object: (2) No such file or directory 2024-09-16T11:52:43.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.382+0000 7f91c03ea640 -1 rbd::mirror::InstanceWatcher: 0x563b43045180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.382+0000 7f91ba3de640 -1 rbd::mirror::MirrorStatusUpdater 0x563b456d4240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.382+0000 7f91c73f8640 -1 rbd::mirror::NamespaceReplayer: 0x563b456164e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.383+0000 7f91c0beb640 -1 rbd::mirror::MirrorStatusUpdater 0x563b44a8b440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.383+0000 7f91c6bf7640 -1 rbd::mirror::NamespaceReplayer: 0x563b456164e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.395 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3' 2024-09-16T11:52:43.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-16T11:52:43.396 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.406 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-16T11:52:43.406 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.435+0000 7fca6dbf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 72309) UID: 1000 2024-09-16T11:52:43.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.435+0000 7fca6cbf7640 -1 rbd::mirror::LeaderWatcher: 0x55cd2b3e3800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.435+0000 7fca6cbf7640 -1 rbd::mirror::InstanceWatcher: 0x55cd280cf500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6556 object: (2) No such file or directory 2024-09-16T11:52:43.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca6d3f8640 -1 rbd::mirror::InstanceWatcher: 0x55cd2c012000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6556 object: (2) No such file or directory 2024-09-16T11:52:43.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca6cbf7640 -1 rbd::mirror::InstanceWatcher: 0x55cd280cf880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6559 object: (2) No such file or directory 2024-09-16T11:52:43.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca66beb640 -1 rbd::mirror::InstanceWatcher: 0x55cd280cf500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca58bcf640 -1 rbd::mirror::InstanceWatcher: 0x55cd280cf880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca603de640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd28125c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca6d3f8640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a7de000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca523c2640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd28125e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.436+0000 7fca6cbf7640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a7deb60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.437+0000 7fca593d0640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd2b371e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.437+0000 7fca6d3f8640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a7deb60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.437+0000 7fca663ea640 -1 rbd::mirror::InstanceWatcher: 0x55cd2c012000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.437+0000 7fca663ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd2a7e0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.437+0000 7fca6d3f8640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a7de000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.438+0000 7fca5fbdd640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd2a7e0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.438+0000 7fca6cbf7640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a7de1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.438+0000 7fca66beb640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd2a7e0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.438+0000 7fca6d3f8640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a7de1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.439+0000 7fca6d3f8640 -1 rbd::mirror::LeaderWatcher: 0x55cd2a6d0600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.439+0000 7fca6d3f8640 -1 rbd::mirror::InstanceWatcher: 0x55cd280cf180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6556 object: (2) No such file or directory 2024-09-16T11:52:43.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.439+0000 7fca663ea640 -1 rbd::mirror::InstanceWatcher: 0x55cd280cf180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.440+0000 7fca5fbdd640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd2a76c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.440+0000 7fca6d3f8640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a6b44e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.440+0000 7fca663ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55cd29b15560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.440+0000 7fca6cbf7640 -1 rbd::mirror::NamespaceReplayer: 0x55cd2a6b44e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.453 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2' 2024-09-16T11:52:43.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-16T11:52:43.454 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.463 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-16T11:52:43.463 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.493+0000 7fd9c1df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 71780) UID: 1000 2024-09-16T11:52:43.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.493+0000 7fd9c0df7640 -1 rbd::mirror::LeaderWatcher: 0x55fbd99f1800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.493+0000 7fd9c0df7640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6511 object: (2) No such file or directory 2024-09-16T11:52:43.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c0df7640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6511 object: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c0df7640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6514 object: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9badeb640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9badeb640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9ad5d0640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9b45de640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd9814360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9b45de640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd9814480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c15f8640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd9812000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9a65c2640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd99fe6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c15f8640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd98121a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c15f8640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd9812b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9ba5ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd9814000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9ba5ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd9814120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c0df7640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd9812000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c0df7640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd98121a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9acdcf640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd9969e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.494+0000 7fd9c15f8640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd9812b60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.496+0000 7fd9c0df7640 -1 rbd::mirror::LeaderWatcher: 0x55fbd8c70600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.497+0000 7fd9c0df7640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6511 object: (2) No such file or directory 2024-09-16T11:52:43.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.497+0000 7fd9badeb640 -1 rbd::mirror::InstanceWatcher: 0x55fbd6691180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.497+0000 7fd9b3ddd640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd8c8e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.497+0000 7fd9c0df7640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd8c6a4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.497+0000 7fd9badeb640 -1 rbd::mirror::MirrorStatusUpdater 0x55fbd80a90e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.497+0000 7fd9c15f8640 -1 rbd::mirror::NamespaceReplayer: 0x55fbd8c6a4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.510 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2024-09-16T11:52:43.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-16T11:52:43.510 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.524 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-16T11:52:43.525 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.550+0000 7f78533f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 73991) UID: 1000 2024-09-16T11:52:43.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.550+0000 7f78523f7640 -1 rbd::mirror::LeaderWatcher: 0x560b968ed800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f784c3eb640 -1 rbd::mirror::InstanceWatcher: 0x560b935eb500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f784c3eb640 -1 rbd::mirror::InstanceWatcher: 0x560b9753a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f7852bf8640 -1 rbd::mirror::InstanceWatcher: 0x560b935eb880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6708 object: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f783e3cf640 -1 rbd::mirror::InstanceWatcher: 0x560b935eb880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f7845bde640 -1 rbd::mirror::MirrorStatusUpdater 0x560b93641c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f7852bf8640 -1 rbd::mirror::NamespaceReplayer: 0x560b95cfc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f78453dd640 -1 rbd::mirror::MirrorStatusUpdater 0x560b95cfe360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f7852bf8640 -1 rbd::mirror::NamespaceReplayer: 0x560b95cfc1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f7837bc2640 -1 rbd::mirror::MirrorStatusUpdater 0x560b93641e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f78523f7640 -1 rbd::mirror::NamespaceReplayer: 0x560b9750a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.551+0000 7f783ebd0640 -1 rbd::mirror::MirrorStatusUpdater 0x560b96906000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.552+0000 7f7852bf8640 -1 rbd::mirror::NamespaceReplayer: 0x560b9750a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.554+0000 7f784c3eb640 -1 rbd::mirror::InstanceWatcher: 0x560b935eb180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.554+0000 7f78453dd640 -1 rbd::mirror::MirrorStatusUpdater 0x560b95c8e240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.554+0000 7f78523f7640 -1 rbd::mirror::NamespaceReplayer: 0x560b95bd04e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.566 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0' 2024-09-16T11:52:43.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-16T11:52:43.567 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.578 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-16T11:52:43.578 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.606+0000 7fe7d77f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 60853) UID: 1000 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7d07eb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555d05a88000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7d67f7640 -1 rbd::mirror::LeaderWatcher: 0x555d04115800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7d07eb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555d05a88b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7cffea640 -1 rbd::mirror::PoolWatcher: 0x555d058023c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7cffea640 -1 rbd::mirror::PoolWatcher: 0x555d058025a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7c97dd640 -1 rbd::mirror::PoolWatcher: 0x555d05116f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7c97dd640 -1 rbd::mirror::PoolWatcher: 0x555d05802000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7d67f7640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5388 object: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7d67f7640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5385 object: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7d67f7640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5385 object: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.607+0000 7fe7c27cf640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7cffea640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7cffea640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7bc7c3640 -1 rbd::mirror::MirrorStatusUpdater 0x555d041186c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d67f7640 -1 rbd::mirror::NamespaceReplayer: 0x555d04c32340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7c9fde640 -1 rbd::mirror::MirrorStatusUpdater 0x555d03f66360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7c9fde640 -1 rbd::mirror::MirrorStatusUpdater 0x555d00e29c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d6ff8640 -1 rbd::mirror::NamespaceReplayer: 0x555d03f64000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d6ff8640 -1 rbd::mirror::NamespaceReplayer: 0x555d03f641a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7c27cf640 -1 rbd::mirror::MirrorStatusUpdater 0x555d0409be60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d6ff8640 -1 rbd::mirror::NamespaceReplayer: 0x555d04c32340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d07eb640 -1 rbd::mirror::MirrorStatusUpdater 0x555d03f66000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d07eb640 -1 rbd::mirror::MirrorStatusUpdater 0x555d03f66120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d6ff8640 -1 rbd::mirror::NamespaceReplayer: 0x555d03f64000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.608+0000 7fe7d6ff8640 -1 rbd::mirror::NamespaceReplayer: 0x555d03f641a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.610+0000 7fe7cffea640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555d05a89200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T11:52:43.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.610+0000 7fe7c9fde640 -1 rbd::mirror::PoolWatcher: 0x555d053825a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.610+0000 7fe7cffea640 -1 rbd::mirror::PoolWatcher: 0x555d05117e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T11:52:43.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.611+0000 7fe7d67f7640 -1 rbd::mirror::LeaderWatcher: 0x555d033ce600 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T11:52:43.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.611+0000 7fe7d67f7640 -1 rbd::mirror::LeaderWatcher: 0x555d033ce600 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-16T11:52:43.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.611+0000 7fe7d6ff8640 -1 rbd::mirror::LeaderWatcher: 0x555d033ce600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.611+0000 7fe7d6ff8640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5385 object: (2) No such file or directory 2024-09-16T11:52:43.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.612+0000 7fe7d07eb640 -1 rbd::mirror::InstanceWatcher: 0x555d00dd3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.612+0000 7fe7c97dd640 -1 rbd::mirror::MirrorStatusUpdater 0x555d0346a240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.612+0000 7fe7d6ff8640 -1 rbd::mirror::NamespaceReplayer: 0x555d033b84e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.612+0000 7fe7cffea640 -1 rbd::mirror::MirrorStatusUpdater 0x555d02803440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:2024-09-16T11:52:43.612+0000 7fe7d67f7640 -1 rbd::mirror::NamespaceReplayer: 0x555d033b84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.627 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3' 2024-09-16T11:52:43.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-16T11:52:43.628 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.638 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-16T11:52:43.638 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.667+0000 7f405d9f9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 62514) UID: 1000 2024-09-16T11:52:43.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.667+0000 7f405c9f7640 -1 rbd::mirror::LeaderWatcher: 0x55eecb6ef800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.668+0000 7f405c9f7640 -1 rbd::mirror::InstanceWatcher: 0x55eecc306000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5502 object: (2) No such file or directory 2024-09-16T11:52:43.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.668+0000 7f40491d0640 -1 rbd::mirror::InstanceWatcher: 0x55eecc306000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.668+0000 7f40421c2640 -1 rbd::mirror::MirrorStatusUpdater 0x55eecc2345a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.668+0000 7f405c9f7640 -1 rbd::mirror::NamespaceReplayer: 0x55eecb55cb60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.669+0000 7f40489cf640 -1 rbd::mirror::MirrorStatusUpdater 0x55eecb691e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:2024-09-16T11:52:43.669+0000 7f405d1f8640 -1 rbd::mirror::NamespaceReplayer: 0x55eecb55cb60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.683 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2' 2024-09-16T11:52:43.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-16T11:52:43.683 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.693 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-16T11:52:43.693 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.722+0000 7f7decbf9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 73779) UID: 1000 2024-09-16T11:52:43.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.723+0000 7f7dec3f8640 -1 rbd::mirror::InstanceWatcher: 0x561277b0d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6175 object: (2) No such file or directory 2024-09-16T11:52:43.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.723+0000 7f7debbf7640 -1 rbd::mirror::InstanceWatcher: 0x561277b0d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6175 object: (2) No such file or directory 2024-09-16T11:52:43.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.723+0000 7f7dd83d0640 -1 rbd::mirror::InstanceWatcher: 0x561277b0dc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.723+0000 7f7de5beb640 -1 rbd::mirror::InstanceWatcher: 0x561277b0d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.723+0000 7f7de5beb640 -1 rbd::mirror::InstanceWatcher: 0x561277b0d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.723+0000 7f7dd1bc3640 -1 rbd::mirror::MirrorStatusUpdater 0x56127b97e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.724+0000 7f7dec3f8640 -1 rbd::mirror::NamespaceReplayer: 0x56127b96c340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.724+0000 7f7de53ea640 -1 rbd::mirror::MirrorStatusUpdater 0x56127aca4000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.724+0000 7f7debbf7640 -1 rbd::mirror::NamespaceReplayer: 0x56127aca8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.724+0000 7f7de5beb640 -1 rbd::mirror::MirrorStatusUpdater 0x56127aca4120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.724+0000 7f7dec3f8640 -1 rbd::mirror::NamespaceReplayer: 0x56127aca81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.725+0000 7f7dec3f8640 -1 rbd::mirror::LeaderWatcher: 0x56127a0fe600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.726+0000 7f7dec3f8640 -1 rbd::mirror::InstanceWatcher: 0x561277b0d180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6175 object: (2) No such file or directory 2024-09-16T11:52:43.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.726+0000 7f7de53ea640 -1 rbd::mirror::InstanceWatcher: 0x561277b0d180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.726+0000 7f7de5beb640 -1 rbd::mirror::MirrorStatusUpdater 0x561279535440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:2024-09-16T11:52:43.726+0000 7f7dec3f8640 -1 rbd::mirror::NamespaceReplayer: 0x56127a0f24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.739 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1' 2024-09-16T11:52:43.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-16T11:52:43.740 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T11:52:43.753 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-16T11:52:43.753 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:52:43.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.779+0000 7f9da9df9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72917) UID: 1000 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da8df7640 -1 rbd::mirror::InstanceWatcher: 0x55ae68642000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6129 object: (2) No such file or directory 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da8df7640 -1 rbd::mirror::InstanceWatcher: 0x55ae64701880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6129 object: (2) No such file or directory 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9d94dcf640 -1 rbd::mirror::InstanceWatcher: 0x55ae64701500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da25ea640 -1 rbd::mirror::InstanceWatcher: 0x55ae68642000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da25ea640 -1 rbd::mirror::InstanceWatcher: 0x55ae64701880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9d8edc3640 -1 rbd::mirror::MirrorStatusUpdater 0x55ae64757c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da8df7640 -1 rbd::mirror::NamespaceReplayer: 0x55ae6854c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da25ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55ae67a126c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da25ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55ae67a127e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da8df7640 -1 rbd::mirror::NamespaceReplayer: 0x55ae68622000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.780+0000 7f9da8df7640 -1 rbd::mirror::NamespaceReplayer: 0x55ae686221a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.782+0000 7f9da8df7640 -1 rbd::mirror::LeaderWatcher: 0x55ae66cde600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T11:52:43.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.783+0000 7f9da8df7640 -1 rbd::mirror::InstanceWatcher: 0x55ae64701180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6129 object: (2) No such file or directory 2024-09-16T11:52:43.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.783+0000 7f9da2deb640 -1 rbd::mirror::InstanceWatcher: 0x55ae64701180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T11:52:43.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.783+0000 7f9da25ea640 -1 rbd::mirror::MirrorStatusUpdater 0x55ae66139440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T11:52:43.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:2024-09-16T11:52:43.783+0000 7f9da8df7640 -1 rbd::mirror::NamespaceReplayer: 0x55ae66cda4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T11:52:43.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi103.stderr:daemon-helper: command failed with exit status 1 2024-09-16T11:52:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:52:43.795 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0' 2024-09-16T11:52:43.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-16T11:52:43.796 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T11:52:43.807 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2024-09-16T11:52:43.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:52:44.095 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:52:44.095 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":453,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:52:42.729322+0000","last_up_change":"2024-09-16T11:15:45.243714+0000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:45.620615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:51.517749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":32440},{"type":"v1","addr":"172.21.15.103:6801","nonce":32440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":32440},{"type":"v1","addr":"172.21.15.103:6803","nonce":32440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":32440},{"type":"v1","addr":"172.21.15.103:6807","nonce":32440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":32440},{"type":"v1","addr":"172.21.15.103:6805","nonce":32440}]},"public_addr":"172.21.15.103:6801/32440","cluster_addr":"172.21.15.103:6803/32440","heartbeat_back_addr":"172.21.15.103:6807/32440","heartbeat_front_addr":"172.21.15.103:6805/32440","state":["exists","up"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":32441},{"type":"v1","addr":"172.21.15.103:6815","nonce":32441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":32441},{"type":"v1","addr":"172.21.15.103:6819","nonce":32441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":32441},{"type":"v1","addr":"172.21.15.103:6823","nonce":32441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":32441},{"type":"v1","addr":"172.21.15.103:6821","nonce":32441}]},"public_addr":"172.21.15.103:6815/32441","cluster_addr":"172.21.15.103:6819/32441","heartbeat_back_addr":"172.21.15.103:6823/32441","heartbeat_front_addr":"172.21.15.103:6821/32441","state":["exists","up"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":32442},{"type":"v1","addr":"172.21.15.103:6809","nonce":32442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":32442},{"type":"v1","addr":"172.21.15.103:6811","nonce":32442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":32442},{"type":"v1","addr":"172.21.15.103:6818","nonce":32442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":32442},{"type":"v1","addr":"172.21.15.103:6814","nonce":32442}]},"public_addr":"172.21.15.103:6809/32442","cluster_addr":"172.21.15.103:6811/32442","heartbeat_back_addr":"172.21.15.103:6818/32442","heartbeat_front_addr":"172.21.15.103:6814/32442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.330275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.341245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.390116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2421293078":"2024-09-16T12:52:37.941243+0000","172.21.15.103:0/3805238318":"2024-09-16T12:51:15.721280+0000","172.21.15.103:0/2832263779":"2024-09-16T12:50:32.561028+0000","172.21.15.103:0/2289668829":"2024-09-16T12:48:41.797074+0000","172.21.15.103:0/3771179645":"2024-09-16T12:38:45.409499+0000","172.21.15.103:0/1136218023":"2024-09-16T12:23:33.441004+0000","172.21.15.103:0/646465485":"2024-09-16T12:26:03.664001+0000","172.21.15.103:0/2232440296":"2024-09-16T12:38:31.343976+0000","172.21.15.103:0/1578254804":"2024-09-16T12:38:10.273772+0000","172.21.15.103:0/3597209068":"2024-09-16T12:38:24.340156+0000","172.21.15.103:0/2565862713":"2024-09-16T12:51:56.073730+0000","172.21.15.103:0/2274801631":"2024-09-16T12:37:49.251241+0000","172.21.15.103:0/4084672868":"2024-09-16T12:37:42.403123+0000","172.21.15.103:0/1937224043":"2024-09-16T12:36:31.087626+0000","172.21.15.103:0/216826767":"2024-09-16T12:44:04.549984+0000","172.21.15.103:0/245262417":"2024-09-16T12:43:15.609542+0000","172.21.15.103:0/2417799788":"2024-09-16T12:46:36.764686+0000","172.21.15.103:0/586593529":"2024-09-16T12:37:20.556829+0000","172.21.15.103:0/85330197":"2024-09-16T12:24:21.293785+0000","172.21.15.103:0/1199326373":"2024-09-16T12:37:12.117631+0000","172.21.15.103:0/1395429353":"2024-09-16T12:37:56.274135+0000","172.21.15.103:0/2358511332":"2024-09-16T12:37:28.578190+0000","172.21.15.103:0/2086805952":"2024-09-16T12:37:03.567445+0000","172.21.15.103:0/522747775":"2024-09-16T12:39:21.674544+0000","172.21.15.103:0/3722853032":"2024-09-16T12:18:24.342357+0000","172.21.15.103:0/3558039446":"2024-09-16T12:38:38.360576+0000","172.21.15.103:0/3762303":"2024-09-16T12:43:39.639834+0000","172.21.15.103:0/873707127":"2024-09-16T12:19:10.637973+0000","172.21.15.103:0/3221087990":"2024-09-16T12:39:37.414327+0000","172.21.15.103:0/2568724629":"2024-09-16T12:20:05.664824+0000","172.21.15.103:0/1343234484":"2024-09-16T12:49:54.131762+0000","172.21.15.103:0/3642385604":"2024-09-16T12:36:47.516616+0000","172.21.15.103:0/2358874724":"2024-09-16T12:40:57.630182+0000","172.21.15.103:0/2796835502":"2024-09-16T12:42:49.369986+0000","172.21.15.103:0/3358018536":"2024-09-16T12:20:48.728845+0000","172.21.15.103:0/3552214281":"2024-09-16T12:22:37.825258+0000","172.21.15.103:0/1471954210":"2024-09-16T12:38:54.052890+0000","172.21.15.103:0/2715091826":"2024-09-16T12:47:19.510480+0000","172.21.15.103:0/3908754705":"2024-09-16T12:38:03.289231+0000","172.21.15.103:0/1679816394":"2024-09-16T12:25:16.878432+0000","172.21.15.103:0/2151552903":"2024-09-16T12:21:44.697974+0000","172.21.15.103:0/2868449353":"2024-09-16T12:37:35.209229+0000","172.21.15.103:0/585228112":"2024-09-16T12:40:49.595705+0000","172.21.15.103:0/79919090":"2024-09-16T12:36:39.464334+0000","172.21.15.103:0/1506346485":"2024-09-16T12:40:33.545888+0000","172.21.15.103:0/1462341518":"2024-09-16T12:49:18.897223+0000","172.21.15.103:0/601098036":"2024-09-16T12:41:49.348530+0000","172.21.15.103:0/1232748297":"2024-09-16T12:39:00.472064+0000","172.21.15.103:0/1414253349":"2024-09-16T12:39:07.491205+0000","172.21.15.103:0/1870543683":"2024-09-16T12:39:14.650552+0000","172.21.15.103:0/963388184":"2024-09-16T12:43:31.651612+0000","172.21.15.103:0/894544521":"2024-09-16T12:43:23.604213+0000","172.21.15.103:0/2501687096":"2024-09-16T12:39:28.520945+0000","172.21.15.103:0/3661160834":"2024-09-16T12:40:09.533233+0000","172.21.15.103:0/3719169469":"2024-09-16T12:40:17.494359+0000","172.21.15.103:0/2423584402":"2024-09-16T12:40:25.583433+0000","172.21.15.103:0/2323561327":"2024-09-16T12:40:41.576895+0000","172.21.15.103:0/1170311901":"2024-09-16T12:41:05.605431+0000","172.21.15.103:0/3671434601":"2024-09-16T12:40:01.513270+0000","172.21.15.103:0/3897467870":"2024-09-16T12:41:12.461593+0000","172.21.15.103:0/2162056233":"2024-09-16T12:41:35.316029+0000","172.21.15.103:0/2879017885":"2024-09-16T12:41:42.317254+0000","172.21.15.103:0/158418185":"2024-09-16T12:41:56.535851+0000","172.21.15.103:0/470419518":"2024-09-16T12:42:58.233254+0000","172.21.15.103:0/3840019612":"2024-09-16T12:48:04.815650+0000","172.21.15.103:0/39954540":"2024-09-16T12:36:55.444687+0000","172.21.15.103:0/2617466616":"2024-09-16T12:43:06.809691+0000","172.21.15.103:0/4227607625":"2024-09-16T12:38:17.330092+0000","172.21.15.103:0/442891346":"2024-09-16T12:43:48.594990+0000","172.21.15.103:0/2428875933":"2024-09-16T12:43:56.570883+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:52:44.107 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T11:52:44.108 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:52:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:52:44.367 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:52:44.377 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1560,"stamp":"2024-09-16T11:52:44.170773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3069948,"kb_used_data":16796,"kb_used_omap":766,"kb_used_meta":3052353,"kb_avail":280045572,"statfs":{"total":289910292480,"available":286766665728,"internally_reserved":0,"allocated":17199104,"data_stored":9000605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":785204,"internal_metadata":3125609676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536801334,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-121809,"num_read_kb":-39181256,"num_write":-461903,"num_write_kb":-170908400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-64512,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708950+0000","last_change":"2024-09-16T11:15:55.281946+0000","last_active":"2024-09-16T11:52:16.708950+0000","last_peered":"2024-09-16T11:52:16.708950+0000","last_clean":"2024-09-16T11:52:16.708950+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:52:16.708950+0000","last_undegraded":"2024-09-16T11:52:16.708950+0000","last_fullsized":"2024-09-16T11:52:16.708950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:39.247359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708795+0000","last_change":"2024-09-16T11:15:55.282001+0000","last_active":"2024-09-16T11:52:16.708795+0000","last_peered":"2024-09-16T11:52:16.708795+0000","last_clean":"2024-09-16T11:52:16.708795+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:52:16.708795+0000","last_undegraded":"2024-09-16T11:52:16.708795+0000","last_fullsized":"2024-09-16T11:52:16.708795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:58.677788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708483+0000","last_change":"2024-09-16T11:15:55.281660+0000","last_active":"2024-09-16T11:52:16.708483+0000","last_peered":"2024-09-16T11:52:16.708483+0000","last_clean":"2024-09-16T11:52:16.708483+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:52:16.708483+0000","last_undegraded":"2024-09-16T11:52:16.708483+0000","last_fullsized":"2024-09-16T11:52:16.708483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:18.641643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.707442+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:52:16.707442+0000","last_peered":"2024-09-16T11:52:16.707442+0000","last_clean":"2024-09-16T11:52:16.707442+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:52:16.707442+0000","last_undegraded":"2024-09-16T11:52:16.707442+0000","last_fullsized":"2024-09-16T11:52:16.707442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:28.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":452,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.120949+0000","last_change":"2024-09-16T11:15:55.279290+0000","last_active":"2024-09-16T11:52:17.120949+0000","last_peered":"2024-09-16T11:52:17.120949+0000","last_clean":"2024-09-16T11:52:17.120949+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:52:17.120949+0000","last_undegraded":"2024-09-16T11:52:17.120949+0000","last_fullsized":"2024-09-16T11:52:17.120949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:09.009156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.320947+0000","last_change":"2024-09-16T11:15:55.276334+0000","last_active":"2024-09-16T11:52:17.320947+0000","last_peered":"2024-09-16T11:52:17.320947+0000","last_clean":"2024-09-16T11:52:17.320947+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:52:17.320947+0000","last_undegraded":"2024-09-16T11:52:17.320947+0000","last_fullsized":"2024-09-16T11:52:17.320947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:08.820896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.321742+0000","last_change":"2024-09-16T11:15:55.276351+0000","last_active":"2024-09-16T11:52:17.321742+0000","last_peered":"2024-09-16T11:52:17.321742+0000","last_clean":"2024-09-16T11:52:17.321742+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:52:17.321742+0000","last_undegraded":"2024-09-16T11:52:17.321742+0000","last_fullsized":"2024-09-16T11:52:17.321742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:33.264611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":450,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708125+0000","last_change":"2024-09-16T11:15:55.282319+0000","last_active":"2024-09-16T11:52:16.708125+0000","last_peered":"2024-09-16T11:52:16.708125+0000","last_clean":"2024-09-16T11:52:16.708125+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:52:16.708125+0000","last_undegraded":"2024-09-16T11:52:16.708125+0000","last_fullsized":"2024-09-16T11:52:16.708125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:46.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":515,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.707504+0000","last_change":"2024-09-16T11:15:48.254170+0000","last_active":"2024-09-16T11:52:16.707504+0000","last_peered":"2024-09-16T11:52:16.707504+0000","last_clean":"2024-09-16T11:52:16.707504+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:52:16.707504+0000","last_undegraded":"2024-09-16T11:52:16.707504+0000","last_fullsized":"2024-09-16T11:52:16.707504+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_clean_scrub_stamp":"2024-09-16T11:15:46.244458+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:31.333661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928484,"kb_used_data":5212,"kb_used_omap":250,"kb_used_meta":923013,"kb_avail":93443356,"statfs":{"total":96636764160,"available":95685996544,"internally_reserved":0,"allocated":5337088,"data_stored":2606623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256482,"internal_metadata":945165854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.438,"15min":0.32900000000000001},"min":{"1min":0.16900000000000001,"5min":0.14499999999999999,"15min":0.13800000000000001},"max":{"1min":1.6539999999999999,"5min":3.8780000000000001,"15min":3.8780000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.439,"15min":0.34699999999999998},"min":{"1min":0.19600000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.4189999999999996,"5min":4.4189999999999996,"15min":4.4189999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40699999999999997,"15min":0.32700000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.16700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.375,"15min":0.34200000000000003},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.151},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.47399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923244,"kb_used_data":5792,"kb_used_omap":231,"kb_used_meta":917208,"kb_avail":93448596,"statfs":{"total":96636764160,"available":95691362304,"internally_reserved":0,"allocated":5931008,"data_stored":3196991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":237092,"internal_metadata":939221468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39700000000000002,"15min":0.34599999999999997},"min":{"1min":0.186,"5min":0.151,"15min":0.14000000000000001},"max":{"1min":2.1960000000000002,"5min":3.181,"15min":4.0309999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46200000000000002,"15min":0.374},"min":{"1min":0.252,"5min":0.20100000000000001,"15min":0.156},"max":{"1min":2.234,"5min":3.2120000000000002,"15min":4.0700000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46700000000000003,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.17399999999999999,"15min":0.14899999999999999},"max":{"1min":2.3250000000000002,"5min":2.6509999999999998,"15min":3.3319999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.37,"15min":0.35199999999999998},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.7410000000000001,"5min":2.7410000000000001,"15min":4.1100000000000003},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218220,"kb_used_data":5792,"kb_used_omap":284,"kb_used_meta":1212131,"kb_avail":93153620,"statfs":{"total":96636764160,"available":95389306880,"internally_reserved":0,"allocated":5931008,"data_stored":3196991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":291630,"internal_metadata":1241222354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.432,"15min":0.33300000000000002},"min":{"1min":0.17100000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.49199999999999999,"5min":6.242,"15min":6.242},"last":0.313},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.52900000000000003,"15min":0.376},"min":{"1min":0.182,"5min":0.16200000000000001,"15min":0.14299999999999999},"max":{"1min":0.47299999999999998,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.66000000000000003,"15min":0.40999999999999998},"min":{"1min":0.224,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":0.57599999999999996,"5min":13.201000000000001,"15min":13.201000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.39200000000000002,"15min":0.33100000000000002},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42999999999999999,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:52:44.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:52:44.633 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:52:44.634 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:52:44.645 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1560,"stamp":"2024-09-16T11:52:44.170773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3069948,"kb_used_data":16796,"kb_used_omap":766,"kb_used_meta":3052353,"kb_avail":280045572,"statfs":{"total":289910292480,"available":286766665728,"internally_reserved":0,"allocated":17199104,"data_stored":9000605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":785204,"internal_metadata":3125609676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536801334,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-121809,"num_read_kb":-39181256,"num_write":-461903,"num_write_kb":-170908400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-64512,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708950+0000","last_change":"2024-09-16T11:15:55.281946+0000","last_active":"2024-09-16T11:52:16.708950+0000","last_peered":"2024-09-16T11:52:16.708950+0000","last_clean":"2024-09-16T11:52:16.708950+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:52:16.708950+0000","last_undegraded":"2024-09-16T11:52:16.708950+0000","last_fullsized":"2024-09-16T11:52:16.708950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:39.247359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708795+0000","last_change":"2024-09-16T11:15:55.282001+0000","last_active":"2024-09-16T11:52:16.708795+0000","last_peered":"2024-09-16T11:52:16.708795+0000","last_clean":"2024-09-16T11:52:16.708795+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:52:16.708795+0000","last_undegraded":"2024-09-16T11:52:16.708795+0000","last_fullsized":"2024-09-16T11:52:16.708795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:58.677788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708483+0000","last_change":"2024-09-16T11:15:55.281660+0000","last_active":"2024-09-16T11:52:16.708483+0000","last_peered":"2024-09-16T11:52:16.708483+0000","last_clean":"2024-09-16T11:52:16.708483+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:52:16.708483+0000","last_undegraded":"2024-09-16T11:52:16.708483+0000","last_fullsized":"2024-09-16T11:52:16.708483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:18.641643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.707442+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:52:16.707442+0000","last_peered":"2024-09-16T11:52:16.707442+0000","last_clean":"2024-09-16T11:52:16.707442+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:52:16.707442+0000","last_undegraded":"2024-09-16T11:52:16.707442+0000","last_fullsized":"2024-09-16T11:52:16.707442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:28.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":452,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.120949+0000","last_change":"2024-09-16T11:15:55.279290+0000","last_active":"2024-09-16T11:52:17.120949+0000","last_peered":"2024-09-16T11:52:17.120949+0000","last_clean":"2024-09-16T11:52:17.120949+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:52:17.120949+0000","last_undegraded":"2024-09-16T11:52:17.120949+0000","last_fullsized":"2024-09-16T11:52:17.120949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:09.009156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.320947+0000","last_change":"2024-09-16T11:15:55.276334+0000","last_active":"2024-09-16T11:52:17.320947+0000","last_peered":"2024-09-16T11:52:17.320947+0000","last_clean":"2024-09-16T11:52:17.320947+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:52:17.320947+0000","last_undegraded":"2024-09-16T11:52:17.320947+0000","last_fullsized":"2024-09-16T11:52:17.320947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:08.820896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.321742+0000","last_change":"2024-09-16T11:15:55.276351+0000","last_active":"2024-09-16T11:52:17.321742+0000","last_peered":"2024-09-16T11:52:17.321742+0000","last_clean":"2024-09-16T11:52:17.321742+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:52:17.321742+0000","last_undegraded":"2024-09-16T11:52:17.321742+0000","last_fullsized":"2024-09-16T11:52:17.321742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:33.264611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":450,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708125+0000","last_change":"2024-09-16T11:15:55.282319+0000","last_active":"2024-09-16T11:52:16.708125+0000","last_peered":"2024-09-16T11:52:16.708125+0000","last_clean":"2024-09-16T11:52:16.708125+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:52:16.708125+0000","last_undegraded":"2024-09-16T11:52:16.708125+0000","last_fullsized":"2024-09-16T11:52:16.708125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:46.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":515,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.707504+0000","last_change":"2024-09-16T11:15:48.254170+0000","last_active":"2024-09-16T11:52:16.707504+0000","last_peered":"2024-09-16T11:52:16.707504+0000","last_clean":"2024-09-16T11:52:16.707504+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:52:16.707504+0000","last_undegraded":"2024-09-16T11:52:16.707504+0000","last_fullsized":"2024-09-16T11:52:16.707504+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_clean_scrub_stamp":"2024-09-16T11:15:46.244458+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:31.333661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928484,"kb_used_data":5212,"kb_used_omap":250,"kb_used_meta":923013,"kb_avail":93443356,"statfs":{"total":96636764160,"available":95685996544,"internally_reserved":0,"allocated":5337088,"data_stored":2606623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256482,"internal_metadata":945165854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.438,"15min":0.32900000000000001},"min":{"1min":0.16900000000000001,"5min":0.14499999999999999,"15min":0.13800000000000001},"max":{"1min":1.6539999999999999,"5min":3.8780000000000001,"15min":3.8780000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.439,"15min":0.34699999999999998},"min":{"1min":0.19600000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.4189999999999996,"5min":4.4189999999999996,"15min":4.4189999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40699999999999997,"15min":0.32700000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.16700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.375,"15min":0.34200000000000003},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.151},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.47399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923244,"kb_used_data":5792,"kb_used_omap":231,"kb_used_meta":917208,"kb_avail":93448596,"statfs":{"total":96636764160,"available":95691362304,"internally_reserved":0,"allocated":5931008,"data_stored":3196991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":237092,"internal_metadata":939221468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39700000000000002,"15min":0.34599999999999997},"min":{"1min":0.186,"5min":0.151,"15min":0.14000000000000001},"max":{"1min":2.1960000000000002,"5min":3.181,"15min":4.0309999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46200000000000002,"15min":0.374},"min":{"1min":0.252,"5min":0.20100000000000001,"15min":0.156},"max":{"1min":2.234,"5min":3.2120000000000002,"15min":4.0700000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46700000000000003,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.17399999999999999,"15min":0.14899999999999999},"max":{"1min":2.3250000000000002,"5min":2.6509999999999998,"15min":3.3319999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.37,"15min":0.35199999999999998},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.7410000000000001,"5min":2.7410000000000001,"15min":4.1100000000000003},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218220,"kb_used_data":5792,"kb_used_omap":284,"kb_used_meta":1212131,"kb_avail":93153620,"statfs":{"total":96636764160,"available":95389306880,"internally_reserved":0,"allocated":5931008,"data_stored":3196991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":291630,"internal_metadata":1241222354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.432,"15min":0.33300000000000002},"min":{"1min":0.17100000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.49199999999999999,"5min":6.242,"15min":6.242},"last":0.313},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.52900000000000003,"15min":0.376},"min":{"1min":0.182,"5min":0.16200000000000001,"15min":0.14299999999999999},"max":{"1min":0.47299999999999998,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.66000000000000003,"15min":0.40999999999999998},"min":{"1min":0.224,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":0.57599999999999996,"5min":13.201000000000001,"15min":13.201000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.39200000000000002,"15min":0.33100000000000002},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42999999999999999,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:52:44.646 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T11:52:44.646 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:52:44.905 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:52:44.906 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:52:44.917 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1560,"stamp":"2024-09-16T11:52:44.170773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3069948,"kb_used_data":16796,"kb_used_omap":766,"kb_used_meta":3052353,"kb_avail":280045572,"statfs":{"total":289910292480,"available":286766665728,"internally_reserved":0,"allocated":17199104,"data_stored":9000605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":785204,"internal_metadata":3125609676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536801334,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-121809,"num_read_kb":-39181256,"num_write":-461903,"num_write_kb":-170908400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-64512,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708950+0000","last_change":"2024-09-16T11:15:55.281946+0000","last_active":"2024-09-16T11:52:16.708950+0000","last_peered":"2024-09-16T11:52:16.708950+0000","last_clean":"2024-09-16T11:52:16.708950+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:52:16.708950+0000","last_undegraded":"2024-09-16T11:52:16.708950+0000","last_fullsized":"2024-09-16T11:52:16.708950+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:39.247359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708795+0000","last_change":"2024-09-16T11:15:55.282001+0000","last_active":"2024-09-16T11:52:16.708795+0000","last_peered":"2024-09-16T11:52:16.708795+0000","last_clean":"2024-09-16T11:52:16.708795+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:52:16.708795+0000","last_undegraded":"2024-09-16T11:52:16.708795+0000","last_fullsized":"2024-09-16T11:52:16.708795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:58.677788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708483+0000","last_change":"2024-09-16T11:15:55.281660+0000","last_active":"2024-09-16T11:52:16.708483+0000","last_peered":"2024-09-16T11:52:16.708483+0000","last_clean":"2024-09-16T11:52:16.708483+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:52:16.708483+0000","last_undegraded":"2024-09-16T11:52:16.708483+0000","last_fullsized":"2024-09-16T11:52:16.708483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:18.641643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.707442+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:52:16.707442+0000","last_peered":"2024-09-16T11:52:16.707442+0000","last_clean":"2024-09-16T11:52:16.707442+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:52:16.707442+0000","last_undegraded":"2024-09-16T11:52:16.707442+0000","last_fullsized":"2024-09-16T11:52:16.707442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:28.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":452,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.120949+0000","last_change":"2024-09-16T11:15:55.279290+0000","last_active":"2024-09-16T11:52:17.120949+0000","last_peered":"2024-09-16T11:52:17.120949+0000","last_clean":"2024-09-16T11:52:17.120949+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:52:17.120949+0000","last_undegraded":"2024-09-16T11:52:17.120949+0000","last_fullsized":"2024-09-16T11:52:17.120949+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:09.009156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.320947+0000","last_change":"2024-09-16T11:15:55.276334+0000","last_active":"2024-09-16T11:52:17.320947+0000","last_peered":"2024-09-16T11:52:17.320947+0000","last_clean":"2024-09-16T11:52:17.320947+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:52:17.320947+0000","last_undegraded":"2024-09-16T11:52:17.320947+0000","last_fullsized":"2024-09-16T11:52:17.320947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:08.820896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:17.321742+0000","last_change":"2024-09-16T11:15:55.276351+0000","last_active":"2024-09-16T11:52:17.321742+0000","last_peered":"2024-09-16T11:52:17.321742+0000","last_clean":"2024-09-16T11:52:17.321742+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:52:17.321742+0000","last_undegraded":"2024-09-16T11:52:17.321742+0000","last_fullsized":"2024-09-16T11:52:17.321742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:33.264611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":450,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.708125+0000","last_change":"2024-09-16T11:15:55.282319+0000","last_active":"2024-09-16T11:52:16.708125+0000","last_peered":"2024-09-16T11:52:16.708125+0000","last_clean":"2024-09-16T11:52:16.708125+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:52:16.708125+0000","last_undegraded":"2024-09-16T11:52:16.708125+0000","last_fullsized":"2024-09-16T11:52:16.708125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:46.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":515,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-16T11:52:16.707504+0000","last_change":"2024-09-16T11:15:48.254170+0000","last_active":"2024-09-16T11:52:16.707504+0000","last_peered":"2024-09-16T11:52:16.707504+0000","last_clean":"2024-09-16T11:52:16.707504+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:52:16.707504+0000","last_undegraded":"2024-09-16T11:52:16.707504+0000","last_fullsized":"2024-09-16T11:52:16.707504+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_clean_scrub_stamp":"2024-09-16T11:15:46.244458+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:31.333661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928484,"kb_used_data":5212,"kb_used_omap":250,"kb_used_meta":923013,"kb_avail":93443356,"statfs":{"total":96636764160,"available":95685996544,"internally_reserved":0,"allocated":5337088,"data_stored":2606623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256482,"internal_metadata":945165854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.438,"15min":0.32900000000000001},"min":{"1min":0.16900000000000001,"5min":0.14499999999999999,"15min":0.13800000000000001},"max":{"1min":1.6539999999999999,"5min":3.8780000000000001,"15min":3.8780000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.439,"15min":0.34699999999999998},"min":{"1min":0.19600000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.4189999999999996,"5min":4.4189999999999996,"15min":4.4189999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40699999999999997,"15min":0.32700000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.16700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.375,"15min":0.34200000000000003},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.151},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.47399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923244,"kb_used_data":5792,"kb_used_omap":231,"kb_used_meta":917208,"kb_avail":93448596,"statfs":{"total":96636764160,"available":95691362304,"internally_reserved":0,"allocated":5931008,"data_stored":3196991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":237092,"internal_metadata":939221468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39700000000000002,"15min":0.34599999999999997},"min":{"1min":0.186,"5min":0.151,"15min":0.14000000000000001},"max":{"1min":2.1960000000000002,"5min":3.181,"15min":4.0309999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46200000000000002,"15min":0.374},"min":{"1min":0.252,"5min":0.20100000000000001,"15min":0.156},"max":{"1min":2.234,"5min":3.2120000000000002,"15min":4.0700000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46700000000000003,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.17399999999999999,"15min":0.14899999999999999},"max":{"1min":2.3250000000000002,"5min":2.6509999999999998,"15min":3.3319999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.37,"15min":0.35199999999999998},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.7410000000000001,"5min":2.7410000000000001,"15min":4.1100000000000003},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218220,"kb_used_data":5792,"kb_used_omap":284,"kb_used_meta":1212131,"kb_avail":93153620,"statfs":{"total":96636764160,"available":95389306880,"internally_reserved":0,"allocated":5931008,"data_stored":3196991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":291630,"internal_metadata":1241222354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.432,"15min":0.33300000000000002},"min":{"1min":0.17100000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.49199999999999999,"5min":6.242,"15min":6.242},"last":0.313},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.52900000000000003,"15min":0.376},"min":{"1min":0.182,"5min":0.16200000000000001,"15min":0.14299999999999999},"max":{"1min":0.47299999999999998,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.66000000000000003,"15min":0.40999999999999998},"min":{"1min":0.224,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":0.57599999999999996,"5min":13.201000000000001,"15min":13.201000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.39200000000000002,"15min":0.33100000000000002},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42999999999999999,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:52:44.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:52:45.178 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:52:45.178 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":453,"fsid":"6599c54d-59b2-4c65-8b6b-ee47e617df73","created":"2024-09-16T11:15:36.443522+0000","modified":"2024-09-16T11:52:42.729322+0000","last_up_change":"2024-09-16T11:15:45.243714+0000","last_in_change":"2024-09-16T11:15:37.248420+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:45.620615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:51.517749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ccbdd57e-4ebb-4834-b138-dce5b7f305da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":32440},{"type":"v1","addr":"172.21.15.103:6801","nonce":32440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":32440},{"type":"v1","addr":"172.21.15.103:6803","nonce":32440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":32440},{"type":"v1","addr":"172.21.15.103:6807","nonce":32440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":32440},{"type":"v1","addr":"172.21.15.103:6805","nonce":32440}]},"public_addr":"172.21.15.103:6801/32440","cluster_addr":"172.21.15.103:6803/32440","heartbeat_back_addr":"172.21.15.103:6807/32440","heartbeat_front_addr":"172.21.15.103:6805/32440","state":["exists","up"]},{"osd":1,"uuid":"e86d8bdd-47fb-438e-b101-4b10cd95fac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6813","nonce":32441},{"type":"v1","addr":"172.21.15.103:6815","nonce":32441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6817","nonce":32441},{"type":"v1","addr":"172.21.15.103:6819","nonce":32441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":32441},{"type":"v1","addr":"172.21.15.103:6823","nonce":32441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":32441},{"type":"v1","addr":"172.21.15.103:6821","nonce":32441}]},"public_addr":"172.21.15.103:6815/32441","cluster_addr":"172.21.15.103:6819/32441","heartbeat_back_addr":"172.21.15.103:6823/32441","heartbeat_front_addr":"172.21.15.103:6821/32441","state":["exists","up"]},{"osd":2,"uuid":"e5c3a350-1028-4bca-a644-b4144e776f7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":32442},{"type":"v1","addr":"172.21.15.103:6809","nonce":32442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":32442},{"type":"v1","addr":"172.21.15.103:6811","nonce":32442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":32442},{"type":"v1","addr":"172.21.15.103:6818","nonce":32442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":32442},{"type":"v1","addr":"172.21.15.103:6814","nonce":32442}]},"public_addr":"172.21.15.103:6809/32442","cluster_addr":"172.21.15.103:6811/32442","heartbeat_back_addr":"172.21.15.103:6818/32442","heartbeat_front_addr":"172.21.15.103:6814/32442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.330275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.341245+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:42.390116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/2421293078":"2024-09-16T12:52:37.941243+0000","172.21.15.103:0/3805238318":"2024-09-16T12:51:15.721280+0000","172.21.15.103:0/2832263779":"2024-09-16T12:50:32.561028+0000","172.21.15.103:0/2289668829":"2024-09-16T12:48:41.797074+0000","172.21.15.103:0/3771179645":"2024-09-16T12:38:45.409499+0000","172.21.15.103:0/1136218023":"2024-09-16T12:23:33.441004+0000","172.21.15.103:0/646465485":"2024-09-16T12:26:03.664001+0000","172.21.15.103:0/2232440296":"2024-09-16T12:38:31.343976+0000","172.21.15.103:0/1578254804":"2024-09-16T12:38:10.273772+0000","172.21.15.103:0/3597209068":"2024-09-16T12:38:24.340156+0000","172.21.15.103:0/2565862713":"2024-09-16T12:51:56.073730+0000","172.21.15.103:0/2274801631":"2024-09-16T12:37:49.251241+0000","172.21.15.103:0/4084672868":"2024-09-16T12:37:42.403123+0000","172.21.15.103:0/1937224043":"2024-09-16T12:36:31.087626+0000","172.21.15.103:0/216826767":"2024-09-16T12:44:04.549984+0000","172.21.15.103:0/245262417":"2024-09-16T12:43:15.609542+0000","172.21.15.103:0/2417799788":"2024-09-16T12:46:36.764686+0000","172.21.15.103:0/586593529":"2024-09-16T12:37:20.556829+0000","172.21.15.103:0/85330197":"2024-09-16T12:24:21.293785+0000","172.21.15.103:0/1199326373":"2024-09-16T12:37:12.117631+0000","172.21.15.103:0/1395429353":"2024-09-16T12:37:56.274135+0000","172.21.15.103:0/2358511332":"2024-09-16T12:37:28.578190+0000","172.21.15.103:0/2086805952":"2024-09-16T12:37:03.567445+0000","172.21.15.103:0/522747775":"2024-09-16T12:39:21.674544+0000","172.21.15.103:0/3722853032":"2024-09-16T12:18:24.342357+0000","172.21.15.103:0/3558039446":"2024-09-16T12:38:38.360576+0000","172.21.15.103:0/3762303":"2024-09-16T12:43:39.639834+0000","172.21.15.103:0/873707127":"2024-09-16T12:19:10.637973+0000","172.21.15.103:0/3221087990":"2024-09-16T12:39:37.414327+0000","172.21.15.103:0/2568724629":"2024-09-16T12:20:05.664824+0000","172.21.15.103:0/1343234484":"2024-09-16T12:49:54.131762+0000","172.21.15.103:0/3642385604":"2024-09-16T12:36:47.516616+0000","172.21.15.103:0/2358874724":"2024-09-16T12:40:57.630182+0000","172.21.15.103:0/2796835502":"2024-09-16T12:42:49.369986+0000","172.21.15.103:0/3358018536":"2024-09-16T12:20:48.728845+0000","172.21.15.103:0/3552214281":"2024-09-16T12:22:37.825258+0000","172.21.15.103:0/1471954210":"2024-09-16T12:38:54.052890+0000","172.21.15.103:0/2715091826":"2024-09-16T12:47:19.510480+0000","172.21.15.103:0/3908754705":"2024-09-16T12:38:03.289231+0000","172.21.15.103:0/1679816394":"2024-09-16T12:25:16.878432+0000","172.21.15.103:0/2151552903":"2024-09-16T12:21:44.697974+0000","172.21.15.103:0/2868449353":"2024-09-16T12:37:35.209229+0000","172.21.15.103:0/585228112":"2024-09-16T12:40:49.595705+0000","172.21.15.103:0/79919090":"2024-09-16T12:36:39.464334+0000","172.21.15.103:0/1506346485":"2024-09-16T12:40:33.545888+0000","172.21.15.103:0/1462341518":"2024-09-16T12:49:18.897223+0000","172.21.15.103:0/601098036":"2024-09-16T12:41:49.348530+0000","172.21.15.103:0/1232748297":"2024-09-16T12:39:00.472064+0000","172.21.15.103:0/1414253349":"2024-09-16T12:39:07.491205+0000","172.21.15.103:0/1870543683":"2024-09-16T12:39:14.650552+0000","172.21.15.103:0/963388184":"2024-09-16T12:43:31.651612+0000","172.21.15.103:0/894544521":"2024-09-16T12:43:23.604213+0000","172.21.15.103:0/2501687096":"2024-09-16T12:39:28.520945+0000","172.21.15.103:0/3661160834":"2024-09-16T12:40:09.533233+0000","172.21.15.103:0/3719169469":"2024-09-16T12:40:17.494359+0000","172.21.15.103:0/2423584402":"2024-09-16T12:40:25.583433+0000","172.21.15.103:0/2323561327":"2024-09-16T12:40:41.576895+0000","172.21.15.103:0/1170311901":"2024-09-16T12:41:05.605431+0000","172.21.15.103:0/3671434601":"2024-09-16T12:40:01.513270+0000","172.21.15.103:0/3897467870":"2024-09-16T12:41:12.461593+0000","172.21.15.103:0/2162056233":"2024-09-16T12:41:35.316029+0000","172.21.15.103:0/2879017885":"2024-09-16T12:41:42.317254+0000","172.21.15.103:0/158418185":"2024-09-16T12:41:56.535851+0000","172.21.15.103:0/470419518":"2024-09-16T12:42:58.233254+0000","172.21.15.103:0/3840019612":"2024-09-16T12:48:04.815650+0000","172.21.15.103:0/39954540":"2024-09-16T12:36:55.444687+0000","172.21.15.103:0/2617466616":"2024-09-16T12:43:06.809691+0000","172.21.15.103:0/4227607625":"2024-09-16T12:38:17.330092+0000","172.21.15.103:0/442891346":"2024-09-16T12:43:48.594990+0000","172.21.15.103:0/2428875933":"2024-09-16T12:43:56.570883+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:52:46.192 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-16T11:52:46.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:52:46.320 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T11:52:46.320 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:52:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T11:52:46.331 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-16T11:52:46.590 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T11:52:46.604 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-16T11:52:46.605 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:52:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T11:52:46.720 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:52:46.720 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T11:52:46.730 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-16T11:52:46.989 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T11:52:47.002 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-16T11:52:47.002 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:52:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-16T11:52:47.117 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:52:47.117 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-16T11:52:47.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-16T11:52:47.386 INFO:teuthology.orchestra.run.smithi103.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T11:52:47.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:52:47.656 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:52:47.657 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:52:47.669 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1561,"stamp":"2024-09-16T11:52:46.171222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3070036,"kb_used_data":16876,"kb_used_omap":766,"kb_used_meta":3052353,"kb_avail":280045484,"statfs":{"total":289910292480,"available":286766575616,"internally_reserved":0,"allocated":17281024,"data_stored":9041719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":785204,"internal_metadata":3125609676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536801334,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-121809,"num_read_kb":-39181256,"num_write":-461903,"num_write_kb":-170908400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-64512,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.445178"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":451,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.735577+0000","last_change":"2024-09-16T11:15:55.281946+0000","last_active":"2024-09-16T11:52:40.735577+0000","last_peered":"2024-09-16T11:52:40.735577+0000","last_clean":"2024-09-16T11:52:40.735577+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:52:40.735577+0000","last_undegraded":"2024-09-16T11:52:40.735577+0000","last_fullsized":"2024-09-16T11:52:40.735577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:39.247359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.735524+0000","last_change":"2024-09-16T11:15:55.282001+0000","last_active":"2024-09-16T11:52:40.735524+0000","last_peered":"2024-09-16T11:52:40.735524+0000","last_clean":"2024-09-16T11:52:40.735524+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:52:40.735524+0000","last_undegraded":"2024-09-16T11:52:40.735524+0000","last_fullsized":"2024-09-16T11:52:40.735524+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:58.677788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":451,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.738806+0000","last_change":"2024-09-16T11:15:55.281660+0000","last_active":"2024-09-16T11:52:40.738806+0000","last_peered":"2024-09-16T11:52:40.738806+0000","last_clean":"2024-09-16T11:52:40.738806+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:52:40.738806+0000","last_undegraded":"2024-09-16T11:52:40.738806+0000","last_fullsized":"2024-09-16T11:52:40.738806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:18.641643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":451,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.734795+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:52:40.734795+0000","last_peered":"2024-09-16T11:52:40.734795+0000","last_clean":"2024-09-16T11:52:40.734795+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:52:40.734795+0000","last_undegraded":"2024-09-16T11:52:40.734795+0000","last_fullsized":"2024-09-16T11:52:40.734795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:28.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":454,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:41.129810+0000","last_change":"2024-09-16T11:15:55.279290+0000","last_active":"2024-09-16T11:52:41.129810+0000","last_peered":"2024-09-16T11:52:41.129810+0000","last_clean":"2024-09-16T11:52:41.129810+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:52:41.129810+0000","last_undegraded":"2024-09-16T11:52:41.129810+0000","last_fullsized":"2024-09-16T11:52:41.129810+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:09.009156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":451,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:41.740018+0000","last_change":"2024-09-16T11:15:55.276334+0000","last_active":"2024-09-16T11:52:41.740018+0000","last_peered":"2024-09-16T11:52:41.740018+0000","last_clean":"2024-09-16T11:52:41.740018+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:52:41.740018+0000","last_undegraded":"2024-09-16T11:52:41.740018+0000","last_fullsized":"2024-09-16T11:52:41.740018+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:08.820896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":451,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:41.740018+0000","last_change":"2024-09-16T11:15:55.276351+0000","last_active":"2024-09-16T11:52:41.740018+0000","last_peered":"2024-09-16T11:52:41.740018+0000","last_clean":"2024-09-16T11:52:41.740018+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:52:41.740018+0000","last_undegraded":"2024-09-16T11:52:41.740018+0000","last_fullsized":"2024-09-16T11:52:41.740018+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:33.264611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":452,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.735612+0000","last_change":"2024-09-16T11:15:55.282319+0000","last_active":"2024-09-16T11:52:40.735612+0000","last_peered":"2024-09-16T11:52:40.735612+0000","last_clean":"2024-09-16T11:52:40.735612+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:52:40.735612+0000","last_undegraded":"2024-09-16T11:52:40.735612+0000","last_fullsized":"2024-09-16T11:52:40.735612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:46.760340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":517,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.733052+0000","last_change":"2024-09-16T11:15:48.254170+0000","last_active":"2024-09-16T11:52:40.733052+0000","last_peered":"2024-09-16T11:52:40.733052+0000","last_clean":"2024-09-16T11:52:40.733052+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:52:40.733052+0000","last_undegraded":"2024-09-16T11:52:40.733052+0000","last_fullsized":"2024-09-16T11:52:40.733052+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:46.244458+0000","last_clean_scrub_stamp":"2024-09-16T11:15:46.244458+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:49:31.333661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928512,"kb_used_data":5236,"kb_used_omap":250,"kb_used_meta":923013,"kb_avail":93443328,"statfs":{"total":96636764160,"available":95685967872,"internally_reserved":0,"allocated":5361664,"data_stored":2620315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256482,"internal_metadata":945165854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.438,"15min":0.32900000000000001},"min":{"1min":0.16900000000000001,"5min":0.14499999999999999,"15min":0.13800000000000001},"max":{"1min":1.6539999999999999,"5min":3.8780000000000001,"15min":3.8780000000000001},"last":0.184},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.439,"15min":0.34699999999999998},"min":{"1min":0.19600000000000001,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":4.4189999999999996,"5min":4.4189999999999996,"15min":4.4189999999999996},"last":0.215}]},{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40699999999999997,"15min":0.32700000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.16700000000000001},"max":{"1min":1.7010000000000001,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.375,"15min":0.34200000000000003},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.151},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923272,"kb_used_data":5820,"kb_used_omap":231,"kb_used_meta":917208,"kb_avail":93448568,"statfs":{"total":96636764160,"available":95691333632,"internally_reserved":0,"allocated":5959680,"data_stored":3210702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":237092,"internal_metadata":939221468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39700000000000002,"15min":0.34599999999999997},"min":{"1min":0.186,"5min":0.151,"15min":0.14000000000000001},"max":{"1min":2.1960000000000002,"5min":3.181,"15min":4.0309999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46200000000000002,"15min":0.374},"min":{"1min":0.252,"5min":0.20100000000000001,"15min":0.156},"max":{"1min":2.234,"5min":3.2120000000000002,"15min":4.0700000000000003},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46700000000000003,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.17399999999999999,"15min":0.14899999999999999},"max":{"1min":2.3250000000000002,"5min":2.6509999999999998,"15min":3.3319999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.37,"15min":0.35199999999999998},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":2.7410000000000001,"5min":2.7410000000000001,"15min":4.1100000000000003},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359738816,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218252,"kb_used_data":5820,"kb_used_omap":284,"kb_used_meta":1212131,"kb_avail":93153588,"statfs":{"total":96636764160,"available":95389274112,"internally_reserved":0,"allocated":5959680,"data_stored":3210702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":291630,"internal_metadata":1241222354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.432,"15min":0.33300000000000002},"min":{"1min":0.17100000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.49199999999999999,"5min":6.242,"15min":6.242},"last":0.376},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.52900000000000003,"15min":0.376},"min":{"1min":0.182,"5min":0.16200000000000001,"15min":0.14299999999999999},"max":{"1min":0.47299999999999998,"5min":9.6630000000000003,"15min":9.6630000000000003},"last":0.501}]},{"osd":2,"last update":"Mon Sep 16 11:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.66000000000000003,"15min":0.40999999999999998},"min":{"1min":0.224,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":0.57599999999999996,"5min":13.201000000000001,"15min":13.201000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.39200000000000002,"15min":0.33100000000000002},"min":{"1min":0.191,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42999999999999999,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:52:47.672 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.672 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.672 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.672 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.672 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.672 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.673 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.673 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.673 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T11:15:46.244458+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:52:47.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:53:07.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:53:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:53:07.946 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:53:07.957 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2024-09-16T11:53:06.175150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3070020,"kb_used_data":16876,"kb_used_omap":766,"kb_used_meta":3052353,"kb_avail":280045500,"statfs":{"total":289910292480,"available":286766592000,"internally_reserved":0,"allocated":17281024,"data_stored":9041719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":785204,"internal_metadata":3125609676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002320"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:54.864622+0000","last_change":"2024-09-16T11:52:54.864572+0000","last_active":"2024-09-16T11:52:54.864622+0000","last_peered":"2024-09-16T11:52:54.864622+0000","last_clean":"2024-09-16T11:52:54.864622+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:52:54.864622+0000","last_undegraded":"2024-09-16T11:52:54.864622+0000","last_fullsized":"2024-09-16T11:52:54.864622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:52:54.864467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:52:54.864467+0000","last_clean_scrub_stamp":"2024-09-16T11:52:54.864467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:32:48.648555+0000","scrub_duration":0.001546855,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:53:02.804218+0000","last_change":"2024-09-16T11:53:02.804180+0000","last_active":"2024-09-16T11:53:02.804218+0000","last_peered":"2024-09-16T11:53:02.804218+0000","last_clean":"2024-09-16T11:53:02.804218+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:53:02.804218+0000","last_undegraded":"2024-09-16T11:53:02.804218+0000","last_fullsized":"2024-09-16T11:53:02.804218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:02.804070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:02.804070+0000","last_clean_scrub_stamp":"2024-09-16T11:53:02.804070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:45.083517+0000","scrub_duration":0.0014876049999999999,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:53:00.844089+0000","last_change":"2024-09-16T11:53:00.844059+0000","last_active":"2024-09-16T11:53:00.844089+0000","last_peered":"2024-09-16T11:53:00.844089+0000","last_clean":"2024-09-16T11:53:00.844089+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:53:00.844089+0000","last_undegraded":"2024-09-16T11:53:00.844089+0000","last_fullsized":"2024-09-16T11:53:00.844089+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:00.843959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:00.843959+0000","last_clean_scrub_stamp":"2024-09-16T11:53:00.843959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:46.490466+0000","scrub_duration":0.0015154770000000001,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:42.779930+0000","last_change":"2024-09-16T11:15:55.281657+0000","last_active":"2024-09-16T11:52:42.779930+0000","last_peered":"2024-09-16T11:52:42.779930+0000","last_clean":"2024-09-16T11:52:42.779930+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:52:42.779930+0000","last_undegraded":"2024-09-16T11:52:42.779930+0000","last_fullsized":"2024-09-16T11:52:42.779930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:52.255522+0000","last_clean_scrub_stamp":"2024-09-16T11:15:52.255522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"453'3","reported_seq":466,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:46.926112+0000","last_change":"2024-09-16T11:52:46.926079+0000","last_active":"2024-09-16T11:52:46.926112+0000","last_peered":"2024-09-16T11:52:46.926112+0000","last_clean":"2024-09-16T11:52:46.926112+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:52:46.926112+0000","last_undegraded":"2024-09-16T11:52:46.926112+0000","last_fullsized":"2024-09-16T11:52:46.926112+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3","last_scrub_stamp":"2024-09-16T11:52:46.925967+0000","last_deep_scrub":"453'3","last_deep_scrub_stamp":"2024-09-16T11:52:46.925967+0000","last_clean_scrub_stamp":"2024-09-16T11:52:46.925967+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:59.191626+0000","scrub_duration":0.0024178630000000001,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:52.799558+0000","last_change":"2024-09-16T11:52:52.799516+0000","last_active":"2024-09-16T11:52:52.799558+0000","last_peered":"2024-09-16T11:52:52.799558+0000","last_clean":"2024-09-16T11:52:52.799558+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:52:52.799558+0000","last_undegraded":"2024-09-16T11:52:52.799558+0000","last_fullsized":"2024-09-16T11:52:52.799558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:52:52.799425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:52:52.799425+0000","last_clean_scrub_stamp":"2024-09-16T11:52:52.799425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:25:38.145510+0000","scrub_duration":0.001488802,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:50.816726+0000","last_change":"2024-09-16T11:52:50.816684+0000","last_active":"2024-09-16T11:52:50.816726+0000","last_peered":"2024-09-16T11:52:50.816726+0000","last_clean":"2024-09-16T11:52:50.816726+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:52:50.816726+0000","last_undegraded":"2024-09-16T11:52:50.816726+0000","last_fullsized":"2024-09-16T11:52:50.816726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:52:50.816566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:52:50.816566+0000","last_clean_scrub_stamp":"2024-09-16T11:52:50.816566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:05:50.655412+0000","scrub_duration":0.0015740070000000001,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"453'2","reported_seq":464,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:48.839480+0000","last_change":"2024-09-16T11:52:48.839440+0000","last_active":"2024-09-16T11:52:48.839480+0000","last_peered":"2024-09-16T11:52:48.839480+0000","last_clean":"2024-09-16T11:52:48.839480+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:52:48.839480+0000","last_undegraded":"2024-09-16T11:52:48.839480+0000","last_fullsized":"2024-09-16T11:52:48.839480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'2","last_scrub_stamp":"2024-09-16T11:52:48.839338+0000","last_deep_scrub":"453'2","last_deep_scrub_stamp":"2024-09-16T11:52:48.839338+0000","last_clean_scrub_stamp":"2024-09-16T11:52:48.839338+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:05.945549+0000","scrub_duration":0.0027041389999999999,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"453'54","reported_seq":530,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:47.838027+0000","last_change":"2024-09-16T11:52:47.838006+0000","last_active":"2024-09-16T11:52:47.838027+0000","last_peered":"2024-09-16T11:52:47.838027+0000","last_clean":"2024-09-16T11:52:47.838027+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:52:47.838027+0000","last_undegraded":"2024-09-16T11:52:47.838027+0000","last_fullsized":"2024-09-16T11:52:47.838027+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'54","last_scrub_stamp":"2024-09-16T11:52:47.837935+0000","last_deep_scrub":"453'54","last_deep_scrub_stamp":"2024-09-16T11:52:47.837935+0000","last_clean_scrub_stamp":"2024-09-16T11:52:47.837935+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:42:18.656036+0000","scrub_duration":0.0043017660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928508,"kb_used_data":5236,"kb_used_omap":250,"kb_used_meta":923013,"kb_avail":93443332,"statfs":{"total":96636764160,"available":95685971968,"internally_reserved":0,"allocated":5361664,"data_stored":2620315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256482,"internal_metadata":945165854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38700000000000001,"15min":0.32500000000000001},"min":{"1min":0.16900000000000001,"5min":0.14499999999999999,"15min":0.13800000000000001},"max":{"1min":0.52200000000000002,"5min":3.8780000000000001,"15min":3.8780000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.44600000000000001,"15min":0.34799999999999998},"min":{"1min":0.20399999999999999,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":0.47299999999999998,"5min":4.4189999999999996,"15min":4.4189999999999996},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40699999999999997,"15min":0.33100000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.16700000000000001},"max":{"1min":0.66400000000000003,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38,"15min":0.34699999999999998},"min":{"1min":0.23100000000000001,"5min":0.223,"15min":0.151},"max":{"1min":0.501,"5min":1.603,"15min":1.603},"last":0.31}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923268,"kb_used_data":5820,"kb_used_omap":231,"kb_used_meta":917208,"kb_avail":93448572,"statfs":{"total":96636764160,"available":95691337728,"internally_reserved":0,"allocated":5959680,"data_stored":3210702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":237092,"internal_metadata":939221468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.183,"5min":0.151,"15min":0.14000000000000001},"max":{"1min":0.45800000000000002,"5min":2.1960000000000002,"15min":4.0309999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41799999999999998,"15min":0.38},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.161},"max":{"1min":0.48299999999999998,"5min":2.234,"15min":4.0700000000000003},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.434,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.183,"15min":0.14899999999999999},"max":{"1min":1.9930000000000001,"5min":2.6509999999999998,"15min":3.3319999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.387,"5min":2.7410000000000001,"15min":4.1100000000000003},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359738820,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218244,"kb_used_data":5820,"kb_used_omap":284,"kb_used_meta":1212131,"kb_avail":93153596,"statfs":{"total":96636764160,"available":95389282304,"internally_reserved":0,"allocated":5959680,"data_stored":3210702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":291630,"internal_metadata":1241222354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.34599999999999997,"15min":0.33600000000000002},"min":{"1min":0.20499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.73499999999999999,"5min":4.0359999999999996,"15min":6.242},"last":0.246},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.433,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.182,"15min":0.14299999999999999},"max":{"1min":1.3740000000000001,"5min":5.1660000000000004,"15min":9.6630000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.56100000000000005,"15min":0.41599999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":1.5580000000000001,"5min":13.201000000000001,"15min":13.201000000000001},"last":0.318},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.32600000000000001,"15min":0.33500000000000002},"min":{"1min":0.214,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":4.4770000000000003},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:07.959 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T11:15:52.255522+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=52, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:07.959 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:53:27.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:53:28.232 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-16T11:53:28.232 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-09-16T11:53:28.244 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-09-16T11:53:28.179649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3070020,"kb_used_data":16876,"kb_used_omap":766,"kb_used_meta":3052353,"kb_avail":280045500,"statfs":{"total":289910292480,"available":286766592000,"internally_reserved":0,"allocated":17281024,"data_stored":9041719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":785204,"internal_metadata":3125609676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002383"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:54.864622+0000","last_change":"2024-09-16T11:52:54.864572+0000","last_active":"2024-09-16T11:52:54.864622+0000","last_peered":"2024-09-16T11:52:54.864622+0000","last_clean":"2024-09-16T11:52:54.864622+0000","last_became_active":"2024-09-16T11:15:53.274002+0000","last_became_peered":"2024-09-16T11:15:53.274002+0000","last_unstale":"2024-09-16T11:52:54.864622+0000","last_undegraded":"2024-09-16T11:52:54.864622+0000","last_fullsized":"2024-09-16T11:52:54.864622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:52:54.864467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:52:54.864467+0000","last_clean_scrub_stamp":"2024-09-16T11:52:54.864467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:32:48.648555+0000","scrub_duration":0.001546855,"objects_trimmed":0,"snaptrim_duration":0.00063214,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:53:02.804218+0000","last_change":"2024-09-16T11:53:02.804180+0000","last_active":"2024-09-16T11:53:02.804218+0000","last_peered":"2024-09-16T11:53:02.804218+0000","last_clean":"2024-09-16T11:53:02.804218+0000","last_became_active":"2024-09-16T11:15:53.273982+0000","last_became_peered":"2024-09-16T11:15:53.273982+0000","last_unstale":"2024-09-16T11:53:02.804218+0000","last_undegraded":"2024-09-16T11:53:02.804218+0000","last_fullsized":"2024-09-16T11:53:02.804218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:02.804070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:02.804070+0000","last_clean_scrub_stamp":"2024-09-16T11:53:02.804070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:45.083517+0000","scrub_duration":0.0014876049999999999,"objects_trimmed":0,"snaptrim_duration":0.00069220100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:53:00.844089+0000","last_change":"2024-09-16T11:53:00.844059+0000","last_active":"2024-09-16T11:53:00.844089+0000","last_peered":"2024-09-16T11:53:00.844089+0000","last_clean":"2024-09-16T11:53:00.844089+0000","last_became_active":"2024-09-16T11:15:53.274063+0000","last_became_peered":"2024-09-16T11:15:53.274063+0000","last_unstale":"2024-09-16T11:53:00.844089+0000","last_undegraded":"2024-09-16T11:53:00.844089+0000","last_fullsized":"2024-09-16T11:53:00.844089+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:00.843959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:00.843959+0000","last_clean_scrub_stamp":"2024-09-16T11:53:00.843959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:46.490466+0000","scrub_duration":0.0015154770000000001,"objects_trimmed":0,"snaptrim_duration":0.00046275400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:53:05.714430+0000","last_change":"2024-09-16T11:53:05.714384+0000","last_active":"2024-09-16T11:53:05.714430+0000","last_peered":"2024-09-16T11:53:05.714430+0000","last_clean":"2024-09-16T11:53:05.714430+0000","last_became_active":"2024-09-16T11:15:53.273962+0000","last_became_peered":"2024-09-16T11:15:53.273962+0000","last_unstale":"2024-09-16T11:53:05.714430+0000","last_undegraded":"2024-09-16T11:53:05.714430+0000","last_fullsized":"2024-09-16T11:53:05.714430+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:05.714276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:05.714276+0000","last_clean_scrub_stamp":"2024-09-16T11:53:05.714276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:29.535226+0000","scrub_duration":0.001540841,"objects_trimmed":0,"snaptrim_duration":0.00047634400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"453'3","reported_seq":466,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:46.926112+0000","last_change":"2024-09-16T11:52:46.926079+0000","last_active":"2024-09-16T11:52:46.926112+0000","last_peered":"2024-09-16T11:52:46.926112+0000","last_clean":"2024-09-16T11:52:46.926112+0000","last_became_active":"2024-09-16T11:15:53.272429+0000","last_became_peered":"2024-09-16T11:15:53.272429+0000","last_unstale":"2024-09-16T11:52:46.926112+0000","last_undegraded":"2024-09-16T11:52:46.926112+0000","last_fullsized":"2024-09-16T11:52:46.926112+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'3","last_scrub_stamp":"2024-09-16T11:52:46.925967+0000","last_deep_scrub":"453'3","last_deep_scrub_stamp":"2024-09-16T11:52:46.925967+0000","last_clean_scrub_stamp":"2024-09-16T11:52:46.925967+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:59.191626+0000","scrub_duration":0.0024178630000000001,"objects_trimmed":0,"snaptrim_duration":0.00079456899999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:52.799558+0000","last_change":"2024-09-16T11:52:52.799516+0000","last_active":"2024-09-16T11:52:52.799558+0000","last_peered":"2024-09-16T11:52:52.799558+0000","last_clean":"2024-09-16T11:52:52.799558+0000","last_became_active":"2024-09-16T11:15:53.274408+0000","last_became_peered":"2024-09-16T11:15:53.274408+0000","last_unstale":"2024-09-16T11:52:52.799558+0000","last_undegraded":"2024-09-16T11:52:52.799558+0000","last_fullsized":"2024-09-16T11:52:52.799558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:52:52.799425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:52:52.799425+0000","last_clean_scrub_stamp":"2024-09-16T11:52:52.799425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:25:38.145510+0000","scrub_duration":0.001488802,"objects_trimmed":0,"snaptrim_duration":0.00029934100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:50.816726+0000","last_change":"2024-09-16T11:52:50.816684+0000","last_active":"2024-09-16T11:52:50.816726+0000","last_peered":"2024-09-16T11:52:50.816726+0000","last_clean":"2024-09-16T11:52:50.816726+0000","last_became_active":"2024-09-16T11:15:53.274575+0000","last_became_peered":"2024-09-16T11:15:53.274575+0000","last_unstale":"2024-09-16T11:52:50.816726+0000","last_undegraded":"2024-09-16T11:52:50.816726+0000","last_fullsized":"2024-09-16T11:52:50.816726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:52:50.816566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:52:50.816566+0000","last_clean_scrub_stamp":"2024-09-16T11:52:50.816566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:05:50.655412+0000","scrub_duration":0.0015740070000000001,"objects_trimmed":0,"snaptrim_duration":0.00029473299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"453'2","reported_seq":464,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:48.839480+0000","last_change":"2024-09-16T11:52:48.839440+0000","last_active":"2024-09-16T11:52:48.839480+0000","last_peered":"2024-09-16T11:52:48.839480+0000","last_clean":"2024-09-16T11:52:48.839480+0000","last_became_active":"2024-09-16T11:15:53.272239+0000","last_became_peered":"2024-09-16T11:15:53.272239+0000","last_unstale":"2024-09-16T11:52:48.839480+0000","last_undegraded":"2024-09-16T11:52:48.839480+0000","last_fullsized":"2024-09-16T11:52:48.839480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'2","last_scrub_stamp":"2024-09-16T11:52:48.839338+0000","last_deep_scrub":"453'2","last_deep_scrub_stamp":"2024-09-16T11:52:48.839338+0000","last_clean_scrub_stamp":"2024-09-16T11:52:48.839338+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:05.945549+0000","scrub_duration":0.0027041389999999999,"objects_trimmed":0,"snaptrim_duration":0.0010849329999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"453'54","reported_seq":530,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-16T11:52:47.838027+0000","last_change":"2024-09-16T11:52:47.838006+0000","last_active":"2024-09-16T11:52:47.838027+0000","last_peered":"2024-09-16T11:52:47.838027+0000","last_clean":"2024-09-16T11:52:47.838027+0000","last_became_active":"2024-09-16T11:15:48.253071+0000","last_became_peered":"2024-09-16T11:15:48.253071+0000","last_unstale":"2024-09-16T11:52:47.838027+0000","last_undegraded":"2024-09-16T11:52:47.838027+0000","last_fullsized":"2024-09-16T11:52:47.838027+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'54","last_scrub_stamp":"2024-09-16T11:52:47.837935+0000","last_deep_scrub":"453'54","last_deep_scrub_stamp":"2024-09-16T11:52:47.837935+0000","last_clean_scrub_stamp":"2024-09-16T11:52:47.837935+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:42:18.656036+0000","scrub_duration":0.0043017660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928508,"kb_used_data":5236,"kb_used_omap":250,"kb_used_meta":923013,"kb_avail":93443332,"statfs":{"total":96636764160,"available":95685971968,"internally_reserved":0,"allocated":5361664,"data_stored":2620315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":256482,"internal_metadata":945165854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.38700000000000001,"15min":0.32500000000000001},"min":{"1min":0.16900000000000001,"5min":0.14499999999999999,"15min":0.13800000000000001},"max":{"1min":0.52200000000000002,"5min":3.8780000000000001,"15min":3.8780000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.44600000000000001,"15min":0.34799999999999998},"min":{"1min":0.20399999999999999,"5min":0.16700000000000001,"15min":0.14299999999999999},"max":{"1min":0.47299999999999998,"5min":4.4189999999999996,"15min":4.4189999999999996},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40699999999999997,"15min":0.33100000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.16700000000000001},"max":{"1min":0.66400000000000003,"5min":3.8439999999999999,"15min":3.8439999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38,"15min":0.34699999999999998},"min":{"1min":0.23100000000000001,"5min":0.223,"15min":0.151},"max":{"1min":0.501,"5min":1.603,"15min":1.603},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923268,"kb_used_data":5820,"kb_used_omap":231,"kb_used_meta":917208,"kb_avail":93448572,"statfs":{"total":96636764160,"available":95691337728,"internally_reserved":0,"allocated":5959680,"data_stored":3210702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":237092,"internal_metadata":939221468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"min":{"1min":0.183,"5min":0.151,"15min":0.14000000000000001},"max":{"1min":0.45800000000000002,"5min":2.1960000000000002,"15min":4.0309999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41799999999999998,"15min":0.38},"min":{"1min":0.249,"5min":0.20100000000000001,"15min":0.161},"max":{"1min":0.48299999999999998,"5min":2.234,"15min":4.0700000000000003},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.434,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.183,"15min":0.14899999999999999},"max":{"1min":1.9930000000000001,"5min":2.6509999999999998,"15min":3.3319999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.19600000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.387,"5min":2.7410000000000001,"15min":4.1100000000000003},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738824,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1218244,"kb_used_data":5820,"kb_used_omap":284,"kb_used_meta":1212131,"kb_avail":93153596,"statfs":{"total":96636764160,"available":95389282304,"internally_reserved":0,"allocated":5959680,"data_stored":3210702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":291630,"internal_metadata":1241222354},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.34599999999999997,"15min":0.33600000000000002},"min":{"1min":0.20499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.73499999999999999,"5min":4.0359999999999996,"15min":6.242},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.433,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.182,"15min":0.14299999999999999},"max":{"1min":1.3740000000000001,"5min":5.1660000000000004,"15min":9.6630000000000003},"last":0.38300000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.56100000000000005,"15min":0.41599999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":1.5580000000000001,"5min":13.201000000000001,"15min":13.201000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.32600000000000001,"15min":0.33500000000000002},"min":{"1min":0.214,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":4.4770000000000003},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:28.245 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-16T11:53:28.517 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T11:53:28.517 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T11:53:28.517 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-16T11:53:28.517 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:28.607 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-16T11:53:28.607 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-16T11:53:28.608 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:28.683 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-16T11:53:28.684 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-16T11:53:28.684 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:28.764 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-16T11:53:28.764 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T11:53:28.764 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-16T11:53:28.764 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:28.792 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-16T11:53:28.793 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T11:53:28.793 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-16T11:53:28.793 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:28.806 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-16T11:53:28.806 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T11:53:28.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T11:53:28.877 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:53:28.877 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-16T11:53:29.097 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:53:29.098 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-16T11:53:29.235 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:53:29.236 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-16T11:53:29.360 INFO:tasks.ceph:Archiving mon data... 2024-09-16T11:53:29.361 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/data/cluster2.mon.a.tgz 2024-09-16T11:53:29.362 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-16T11:53:29.377 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.chtahmjJAu 2024-09-16T11:53:29.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.chtahmjJAu 2024-09-16T11:53:29.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.chtahmjJAu 2024-09-16T11:53:29.634 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.chtahmjJAu is 481KB 2024-09-16T11:53:29.732 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.chtahmjJAu 2024-09-16T11:53:29.747 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T11:53:29.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:53:29.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:53:29.812 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T11:53:29.824 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2024-09-16T11:53:29.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:53:30.109 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:30.109 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":464,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:52:41.644478+0000","last_up_change":"2024-09-16T11:15:08.218847+0000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:09.051608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:14.238771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":32155},{"type":"v1","addr":"172.21.15.1:6801","nonce":32155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":32155},{"type":"v1","addr":"172.21.15.1:6803","nonce":32155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":32155},{"type":"v1","addr":"172.21.15.1:6807","nonce":32155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":32155},{"type":"v1","addr":"172.21.15.1:6805","nonce":32155}]},"public_addr":"172.21.15.1:6801/32155","cluster_addr":"172.21.15.1:6803/32155","heartbeat_back_addr":"172.21.15.1:6807/32155","heartbeat_front_addr":"172.21.15.1:6805/32155","state":["exists","up"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":32156},{"type":"v1","addr":"172.21.15.1:6809","nonce":32156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":32156},{"type":"v1","addr":"172.21.15.1:6811","nonce":32156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":32156},{"type":"v1","addr":"172.21.15.1:6815","nonce":32156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":32156},{"type":"v1","addr":"172.21.15.1:6813","nonce":32156}]},"public_addr":"172.21.15.1:6809/32156","cluster_addr":"172.21.15.1:6811/32156","heartbeat_back_addr":"172.21.15.1:6815/32156","heartbeat_front_addr":"172.21.15.1:6813/32156","state":["exists","up"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":32185},{"type":"v1","addr":"172.21.15.1:6817","nonce":32185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":32185},{"type":"v1","addr":"172.21.15.1:6819","nonce":32185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":32185},{"type":"v1","addr":"172.21.15.1:6823","nonce":32185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":32185},{"type":"v1","addr":"172.21.15.1:6821","nonce":32185}]},"public_addr":"172.21.15.1:6817/32185","cluster_addr":"172.21.15.1:6819/32185","heartbeat_back_addr":"172.21.15.1:6823/32185","heartbeat_front_addr":"172.21.15.1:6821/32185","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.073960+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.045396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.041694+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:53:30.121 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T11:53:30.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:53:30.393 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:53:30.405 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1612,"stamp":"2024-09-16T11:53:29.596755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929384,"kb_used_data":12348,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186136,"statfs":{"total":289910292480,"available":288958603264,"internally_reserved":0,"allocated":12644352,"data_stored":7318931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.648582+0000","last_change":"2024-09-16T11:15:17.304545+0000","last_active":"2024-09-16T11:52:39.648582+0000","last_peered":"2024-09-16T11:52:39.648582+0000","last_clean":"2024-09-16T11:52:39.648582+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:52:39.648582+0000","last_undegraded":"2024-09-16T11:52:39.648582+0000","last_fullsized":"2024-09-16T11:52:39.648582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:52.627160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650998+0000","last_change":"2024-09-16T11:15:17.304329+0000","last_active":"2024-09-16T11:52:39.650998+0000","last_peered":"2024-09-16T11:52:39.650998+0000","last_clean":"2024-09-16T11:52:39.650998+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:52:39.650998+0000","last_undegraded":"2024-09-16T11:52:39.650998+0000","last_fullsized":"2024-09-16T11:52:39.650998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:28.342868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.652894+0000","last_change":"2024-09-16T11:15:17.304273+0000","last_active":"2024-09-16T11:52:39.652894+0000","last_peered":"2024-09-16T11:52:39.652894+0000","last_clean":"2024-09-16T11:52:39.652894+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:52:39.652894+0000","last_undegraded":"2024-09-16T11:52:39.652894+0000","last_fullsized":"2024-09-16T11:52:39.652894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:52.193450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.647326+0000","last_change":"2024-09-16T11:15:17.304918+0000","last_active":"2024-09-16T11:52:39.647326+0000","last_peered":"2024-09-16T11:52:39.647326+0000","last_clean":"2024-09-16T11:52:39.647326+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:52:39.647326+0000","last_undegraded":"2024-09-16T11:52:39.647326+0000","last_fullsized":"2024-09-16T11:52:39.647326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:24:48.683341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.587784+0000","last_change":"2024-09-16T11:15:17.302378+0000","last_active":"2024-09-16T11:52:40.587784+0000","last_peered":"2024-09-16T11:52:40.587784+0000","last_clean":"2024-09-16T11:52:40.587784+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:52:40.587784+0000","last_undegraded":"2024-09-16T11:52:40.587784+0000","last_fullsized":"2024-09-16T11:52:40.587784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.265636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.419165+0000","last_change":"2024-09-16T11:15:17.299727+0000","last_active":"2024-09-16T11:52:40.419165+0000","last_peered":"2024-09-16T11:52:40.419165+0000","last_clean":"2024-09-16T11:52:40.419165+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:52:40.419165+0000","last_undegraded":"2024-09-16T11:52:40.419165+0000","last_fullsized":"2024-09-16T11:52:40.419165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:10.958157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.417493+0000","last_change":"2024-09-16T11:15:17.299646+0000","last_active":"2024-09-16T11:52:40.417493+0000","last_peered":"2024-09-16T11:52:40.417493+0000","last_clean":"2024-09-16T11:52:40.417493+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:52:40.417493+0000","last_undegraded":"2024-09-16T11:52:40.417493+0000","last_fullsized":"2024-09-16T11:52:40.417493+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:25.607887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":465,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.655869+0000","last_change":"2024-09-16T11:15:17.304636+0000","last_active":"2024-09-16T11:52:39.655869+0000","last_peered":"2024-09-16T11:52:39.655869+0000","last_clean":"2024-09-16T11:52:39.655869+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:52:39.655869+0000","last_undegraded":"2024-09-16T11:52:39.655869+0000","last_fullsized":"2024-09-16T11:52:39.655869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:15.280839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":528,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650968+0000","last_change":"2024-09-16T11:15:10.226884+0000","last_active":"2024-09-16T11:52:39.650968+0000","last_peered":"2024-09-16T11:52:39.650968+0000","last_clean":"2024-09-16T11:52:39.650968+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:52:39.650968+0000","last_undegraded":"2024-09-16T11:52:39.650968+0000","last_fullsized":"2024-09-16T11:52:39.650968+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_clean_scrub_stamp":"2024-09-16T11:15:09.219587+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:56.273706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313564,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058276,"statfs":{"total":96636764160,"available":96315674624,"internally_reserved":0,"allocated":3817472,"data_stored":2046306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.16},"max":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":1.482},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.372,"15min":0.376},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":0.51600000000000001,"5min":0.55400000000000005,"15min":1.512},"last":0.371}]},{"osd":1,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32100000000000001,"15min":0.32400000000000001},"min":{"1min":0.25,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.41799999999999998,"5min":0.47699999999999998,"15min":1.4750000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":3.0710000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309992,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061848,"statfs":{"total":96636764160,"available":96319332352,"internally_reserved":0,"allocated":4415488,"data_stored":2636693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.30499999999999999,"15min":0.30299999999999999},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":0.439,"5min":0.51600000000000001,"15min":1.0229999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34699999999999998},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":1.2949999999999999},"last":0.314}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":3.6360000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42399999999999999,"5min":0.50800000000000001,"15min":1.9099999999999999},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305828,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066012,"statfs":{"total":96636764160,"available":96323596288,"internally_reserved":0,"allocated":4411392,"data_stored":2635932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.32000000000000001},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.16},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":2.27},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32900000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":2.9169999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.184},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.4449999999999998},"last":0.374},{"interface":"front","average":{"1min":0.318,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.20399999999999999},"max":{"1min":0.439,"5min":0.49299999999999999,"15min":3.0760000000000001},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:30.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:53:30.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:53:30.686 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1612,"stamp":"2024-09-16T11:53:29.596755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929384,"kb_used_data":12348,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186136,"statfs":{"total":289910292480,"available":288958603264,"internally_reserved":0,"allocated":12644352,"data_stored":7318931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.648582+0000","last_change":"2024-09-16T11:15:17.304545+0000","last_active":"2024-09-16T11:52:39.648582+0000","last_peered":"2024-09-16T11:52:39.648582+0000","last_clean":"2024-09-16T11:52:39.648582+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:52:39.648582+0000","last_undegraded":"2024-09-16T11:52:39.648582+0000","last_fullsized":"2024-09-16T11:52:39.648582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:52.627160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650998+0000","last_change":"2024-09-16T11:15:17.304329+0000","last_active":"2024-09-16T11:52:39.650998+0000","last_peered":"2024-09-16T11:52:39.650998+0000","last_clean":"2024-09-16T11:52:39.650998+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:52:39.650998+0000","last_undegraded":"2024-09-16T11:52:39.650998+0000","last_fullsized":"2024-09-16T11:52:39.650998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:28.342868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.652894+0000","last_change":"2024-09-16T11:15:17.304273+0000","last_active":"2024-09-16T11:52:39.652894+0000","last_peered":"2024-09-16T11:52:39.652894+0000","last_clean":"2024-09-16T11:52:39.652894+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:52:39.652894+0000","last_undegraded":"2024-09-16T11:52:39.652894+0000","last_fullsized":"2024-09-16T11:52:39.652894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:52.193450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.647326+0000","last_change":"2024-09-16T11:15:17.304918+0000","last_active":"2024-09-16T11:52:39.647326+0000","last_peered":"2024-09-16T11:52:39.647326+0000","last_clean":"2024-09-16T11:52:39.647326+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:52:39.647326+0000","last_undegraded":"2024-09-16T11:52:39.647326+0000","last_fullsized":"2024-09-16T11:52:39.647326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:24:48.683341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.587784+0000","last_change":"2024-09-16T11:15:17.302378+0000","last_active":"2024-09-16T11:52:40.587784+0000","last_peered":"2024-09-16T11:52:40.587784+0000","last_clean":"2024-09-16T11:52:40.587784+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:52:40.587784+0000","last_undegraded":"2024-09-16T11:52:40.587784+0000","last_fullsized":"2024-09-16T11:52:40.587784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.265636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.419165+0000","last_change":"2024-09-16T11:15:17.299727+0000","last_active":"2024-09-16T11:52:40.419165+0000","last_peered":"2024-09-16T11:52:40.419165+0000","last_clean":"2024-09-16T11:52:40.419165+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:52:40.419165+0000","last_undegraded":"2024-09-16T11:52:40.419165+0000","last_fullsized":"2024-09-16T11:52:40.419165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:10.958157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.417493+0000","last_change":"2024-09-16T11:15:17.299646+0000","last_active":"2024-09-16T11:52:40.417493+0000","last_peered":"2024-09-16T11:52:40.417493+0000","last_clean":"2024-09-16T11:52:40.417493+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:52:40.417493+0000","last_undegraded":"2024-09-16T11:52:40.417493+0000","last_fullsized":"2024-09-16T11:52:40.417493+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:25.607887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":465,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.655869+0000","last_change":"2024-09-16T11:15:17.304636+0000","last_active":"2024-09-16T11:52:39.655869+0000","last_peered":"2024-09-16T11:52:39.655869+0000","last_clean":"2024-09-16T11:52:39.655869+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:52:39.655869+0000","last_undegraded":"2024-09-16T11:52:39.655869+0000","last_fullsized":"2024-09-16T11:52:39.655869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:15.280839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":528,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650968+0000","last_change":"2024-09-16T11:15:10.226884+0000","last_active":"2024-09-16T11:52:39.650968+0000","last_peered":"2024-09-16T11:52:39.650968+0000","last_clean":"2024-09-16T11:52:39.650968+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:52:39.650968+0000","last_undegraded":"2024-09-16T11:52:39.650968+0000","last_fullsized":"2024-09-16T11:52:39.650968+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_clean_scrub_stamp":"2024-09-16T11:15:09.219587+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:56.273706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313564,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058276,"statfs":{"total":96636764160,"available":96315674624,"internally_reserved":0,"allocated":3817472,"data_stored":2046306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.16},"max":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":1.482},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.372,"15min":0.376},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":0.51600000000000001,"5min":0.55400000000000005,"15min":1.512},"last":0.371}]},{"osd":1,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32100000000000001,"15min":0.32400000000000001},"min":{"1min":0.25,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.41799999999999998,"5min":0.47699999999999998,"15min":1.4750000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":3.0710000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309992,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061848,"statfs":{"total":96636764160,"available":96319332352,"internally_reserved":0,"allocated":4415488,"data_stored":2636693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.30499999999999999,"15min":0.30299999999999999},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":0.439,"5min":0.51600000000000001,"15min":1.0229999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34699999999999998},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":1.2949999999999999},"last":0.314}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":3.6360000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42399999999999999,"5min":0.50800000000000001,"15min":1.9099999999999999},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305828,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066012,"statfs":{"total":96636764160,"available":96323596288,"internally_reserved":0,"allocated":4411392,"data_stored":2635932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.32000000000000001},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.16},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":2.27},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32900000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":2.9169999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.184},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.4449999999999998},"last":0.374},{"interface":"front","average":{"1min":0.318,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.20399999999999999},"max":{"1min":0.439,"5min":0.49299999999999999,"15min":3.0760000000000001},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:30.687 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T11:53:30.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:53:30.956 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:30.956 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:53:30.967 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1612,"stamp":"2024-09-16T11:53:29.596755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929384,"kb_used_data":12348,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186136,"statfs":{"total":289910292480,"available":288958603264,"internally_reserved":0,"allocated":12644352,"data_stored":7318931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.648582+0000","last_change":"2024-09-16T11:15:17.304545+0000","last_active":"2024-09-16T11:52:39.648582+0000","last_peered":"2024-09-16T11:52:39.648582+0000","last_clean":"2024-09-16T11:52:39.648582+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:52:39.648582+0000","last_undegraded":"2024-09-16T11:52:39.648582+0000","last_fullsized":"2024-09-16T11:52:39.648582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:52.627160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650998+0000","last_change":"2024-09-16T11:15:17.304329+0000","last_active":"2024-09-16T11:52:39.650998+0000","last_peered":"2024-09-16T11:52:39.650998+0000","last_clean":"2024-09-16T11:52:39.650998+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:52:39.650998+0000","last_undegraded":"2024-09-16T11:52:39.650998+0000","last_fullsized":"2024-09-16T11:52:39.650998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:28.342868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.652894+0000","last_change":"2024-09-16T11:15:17.304273+0000","last_active":"2024-09-16T11:52:39.652894+0000","last_peered":"2024-09-16T11:52:39.652894+0000","last_clean":"2024-09-16T11:52:39.652894+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:52:39.652894+0000","last_undegraded":"2024-09-16T11:52:39.652894+0000","last_fullsized":"2024-09-16T11:52:39.652894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:52.193450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.647326+0000","last_change":"2024-09-16T11:15:17.304918+0000","last_active":"2024-09-16T11:52:39.647326+0000","last_peered":"2024-09-16T11:52:39.647326+0000","last_clean":"2024-09-16T11:52:39.647326+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:52:39.647326+0000","last_undegraded":"2024-09-16T11:52:39.647326+0000","last_fullsized":"2024-09-16T11:52:39.647326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:24:48.683341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.587784+0000","last_change":"2024-09-16T11:15:17.302378+0000","last_active":"2024-09-16T11:52:40.587784+0000","last_peered":"2024-09-16T11:52:40.587784+0000","last_clean":"2024-09-16T11:52:40.587784+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:52:40.587784+0000","last_undegraded":"2024-09-16T11:52:40.587784+0000","last_fullsized":"2024-09-16T11:52:40.587784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.265636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.419165+0000","last_change":"2024-09-16T11:15:17.299727+0000","last_active":"2024-09-16T11:52:40.419165+0000","last_peered":"2024-09-16T11:52:40.419165+0000","last_clean":"2024-09-16T11:52:40.419165+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:52:40.419165+0000","last_undegraded":"2024-09-16T11:52:40.419165+0000","last_fullsized":"2024-09-16T11:52:40.419165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:10.958157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.417493+0000","last_change":"2024-09-16T11:15:17.299646+0000","last_active":"2024-09-16T11:52:40.417493+0000","last_peered":"2024-09-16T11:52:40.417493+0000","last_clean":"2024-09-16T11:52:40.417493+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:52:40.417493+0000","last_undegraded":"2024-09-16T11:52:40.417493+0000","last_fullsized":"2024-09-16T11:52:40.417493+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:25.607887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":465,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.655869+0000","last_change":"2024-09-16T11:15:17.304636+0000","last_active":"2024-09-16T11:52:39.655869+0000","last_peered":"2024-09-16T11:52:39.655869+0000","last_clean":"2024-09-16T11:52:39.655869+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:52:39.655869+0000","last_undegraded":"2024-09-16T11:52:39.655869+0000","last_fullsized":"2024-09-16T11:52:39.655869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:15.280839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":528,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650968+0000","last_change":"2024-09-16T11:15:10.226884+0000","last_active":"2024-09-16T11:52:39.650968+0000","last_peered":"2024-09-16T11:52:39.650968+0000","last_clean":"2024-09-16T11:52:39.650968+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:52:39.650968+0000","last_undegraded":"2024-09-16T11:52:39.650968+0000","last_fullsized":"2024-09-16T11:52:39.650968+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_clean_scrub_stamp":"2024-09-16T11:15:09.219587+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:56.273706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313564,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058276,"statfs":{"total":96636764160,"available":96315674624,"internally_reserved":0,"allocated":3817472,"data_stored":2046306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.16},"max":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":1.482},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.372,"15min":0.376},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":0.51600000000000001,"5min":0.55400000000000005,"15min":1.512},"last":0.371}]},{"osd":1,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32100000000000001,"15min":0.32400000000000001},"min":{"1min":0.25,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.41799999999999998,"5min":0.47699999999999998,"15min":1.4750000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":3.0710000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309992,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061848,"statfs":{"total":96636764160,"available":96319332352,"internally_reserved":0,"allocated":4415488,"data_stored":2636693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.30499999999999999,"15min":0.30299999999999999},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":0.439,"5min":0.51600000000000001,"15min":1.0229999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34699999999999998},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":1.2949999999999999},"last":0.314}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":3.6360000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42399999999999999,"5min":0.50800000000000001,"15min":1.9099999999999999},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305828,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066012,"statfs":{"total":96636764160,"available":96323596288,"internally_reserved":0,"allocated":4411392,"data_stored":2635932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.32000000000000001},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.16},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":2.27},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32900000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":2.9169999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.184},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.4449999999999998},"last":0.374},{"interface":"front","average":{"1min":0.318,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.20399999999999999},"max":{"1min":0.439,"5min":0.49299999999999999,"15min":3.0760000000000001},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:30.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:53:31.242 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:31.242 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":464,"fsid":"36663229-3ca4-47f4-b225-543cee2c8a74","created":"2024-09-16T11:14:58.996735+0000","modified":"2024-09-16T11:52:41.644478+0000","last_up_change":"2024-09-16T11:15:08.218847+0000","last_in_change":"2024-09-16T11:14:59.918514+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:15:09.051608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:15:14.238771+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8fd11c13-6bd9-4aea-9ff4-340755cd147b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":32155},{"type":"v1","addr":"172.21.15.1:6801","nonce":32155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":32155},{"type":"v1","addr":"172.21.15.1:6803","nonce":32155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":32155},{"type":"v1","addr":"172.21.15.1:6807","nonce":32155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":32155},{"type":"v1","addr":"172.21.15.1:6805","nonce":32155}]},"public_addr":"172.21.15.1:6801/32155","cluster_addr":"172.21.15.1:6803/32155","heartbeat_back_addr":"172.21.15.1:6807/32155","heartbeat_front_addr":"172.21.15.1:6805/32155","state":["exists","up"]},{"osd":1,"uuid":"248e1432-8f0f-4e47-9dae-7b8d65418695","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":32156},{"type":"v1","addr":"172.21.15.1:6809","nonce":32156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":32156},{"type":"v1","addr":"172.21.15.1:6811","nonce":32156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":32156},{"type":"v1","addr":"172.21.15.1:6815","nonce":32156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":32156},{"type":"v1","addr":"172.21.15.1:6813","nonce":32156}]},"public_addr":"172.21.15.1:6809/32156","cluster_addr":"172.21.15.1:6811/32156","heartbeat_back_addr":"172.21.15.1:6815/32156","heartbeat_front_addr":"172.21.15.1:6813/32156","state":["exists","up"]},{"osd":2,"uuid":"f66df91e-afd9-425e-bd74-2d5eb60bd0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":32185},{"type":"v1","addr":"172.21.15.1:6817","nonce":32185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":32185},{"type":"v1","addr":"172.21.15.1:6819","nonce":32185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":32185},{"type":"v1","addr":"172.21.15.1:6823","nonce":32185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":32185},{"type":"v1","addr":"172.21.15.1:6821","nonce":32185}]},"public_addr":"172.21.15.1:6817/32185","cluster_addr":"172.21.15.1:6819/32185","heartbeat_back_addr":"172.21.15.1:6823/32185","heartbeat_front_addr":"172.21.15.1:6821/32185","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.073960+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.045396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:15:05.041694+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:53:32.255 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-16T11:53:32.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:53:32.389 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-16T11:53:32.389 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:53:32.389 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-16T11:53:32.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-16T11:53:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T11:53:32.684 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-16T11:53:32.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:53:32.804 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-16T11:53:32.805 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:53:32.805 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-16T11:53:32.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-16T11:53:33.087 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T11:53:33.098 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-16T11:53:33.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:53:33.218 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-09-16T11:53:33.218 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:53:33.218 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-09-16T11:53:33.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-16T11:53:33.501 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T11:53:33.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:53:33.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:33.785 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:53:33.796 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2024-09-16T11:53:33.597531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929384,"kb_used_data":12348,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186136,"statfs":{"total":289910292480,"available":288958603264,"internally_reserved":0,"allocated":12644352,"data_stored":7318931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002518"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.648582+0000","last_change":"2024-09-16T11:15:17.304545+0000","last_active":"2024-09-16T11:52:39.648582+0000","last_peered":"2024-09-16T11:52:39.648582+0000","last_clean":"2024-09-16T11:52:39.648582+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:52:39.648582+0000","last_undegraded":"2024-09-16T11:52:39.648582+0000","last_fullsized":"2024-09-16T11:52:39.648582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:52.627160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650998+0000","last_change":"2024-09-16T11:15:17.304329+0000","last_active":"2024-09-16T11:52:39.650998+0000","last_peered":"2024-09-16T11:52:39.650998+0000","last_clean":"2024-09-16T11:52:39.650998+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:52:39.650998+0000","last_undegraded":"2024-09-16T11:52:39.650998+0000","last_fullsized":"2024-09-16T11:52:39.650998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:28.342868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.652894+0000","last_change":"2024-09-16T11:15:17.304273+0000","last_active":"2024-09-16T11:52:39.652894+0000","last_peered":"2024-09-16T11:52:39.652894+0000","last_clean":"2024-09-16T11:52:39.652894+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:52:39.652894+0000","last_undegraded":"2024-09-16T11:52:39.652894+0000","last_fullsized":"2024-09-16T11:52:39.652894+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:52.193450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.647326+0000","last_change":"2024-09-16T11:15:17.304918+0000","last_active":"2024-09-16T11:52:39.647326+0000","last_peered":"2024-09-16T11:52:39.647326+0000","last_clean":"2024-09-16T11:52:39.647326+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:52:39.647326+0000","last_undegraded":"2024-09-16T11:52:39.647326+0000","last_fullsized":"2024-09-16T11:52:39.647326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:24:48.683341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.587784+0000","last_change":"2024-09-16T11:15:17.302378+0000","last_active":"2024-09-16T11:52:40.587784+0000","last_peered":"2024-09-16T11:52:40.587784+0000","last_clean":"2024-09-16T11:52:40.587784+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:52:40.587784+0000","last_undegraded":"2024-09-16T11:52:40.587784+0000","last_fullsized":"2024-09-16T11:52:40.587784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.265636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.419165+0000","last_change":"2024-09-16T11:15:17.299727+0000","last_active":"2024-09-16T11:52:40.419165+0000","last_peered":"2024-09-16T11:52:40.419165+0000","last_clean":"2024-09-16T11:52:40.419165+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:52:40.419165+0000","last_undegraded":"2024-09-16T11:52:40.419165+0000","last_fullsized":"2024-09-16T11:52:40.419165+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:10.958157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:40.417493+0000","last_change":"2024-09-16T11:15:17.299646+0000","last_active":"2024-09-16T11:52:40.417493+0000","last_peered":"2024-09-16T11:52:40.417493+0000","last_clean":"2024-09-16T11:52:40.417493+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:52:40.417493+0000","last_undegraded":"2024-09-16T11:52:40.417493+0000","last_fullsized":"2024-09-16T11:52:40.417493+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:25.607887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":465,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.655869+0000","last_change":"2024-09-16T11:15:17.304636+0000","last_active":"2024-09-16T11:52:39.655869+0000","last_peered":"2024-09-16T11:52:39.655869+0000","last_clean":"2024-09-16T11:52:39.655869+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:52:39.655869+0000","last_undegraded":"2024-09-16T11:52:39.655869+0000","last_fullsized":"2024-09-16T11:52:39.655869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:14.288946+0000","last_clean_scrub_stamp":"2024-09-16T11:15:14.288946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:15.280839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":528,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T11:52:39.650968+0000","last_change":"2024-09-16T11:15:10.226884+0000","last_active":"2024-09-16T11:52:39.650968+0000","last_peered":"2024-09-16T11:52:39.650968+0000","last_clean":"2024-09-16T11:52:39.650968+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:52:39.650968+0000","last_undegraded":"2024-09-16T11:52:39.650968+0000","last_fullsized":"2024-09-16T11:52:39.650968+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:15:09.219587+0000","last_clean_scrub_stamp":"2024-09-16T11:15:09.219587+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:56.273706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313564,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058276,"statfs":{"total":96636764160,"available":96315674624,"internally_reserved":0,"allocated":3817472,"data_stored":2046306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.16},"max":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":1.482},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.372,"15min":0.376},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":0.51600000000000001,"5min":0.55400000000000005,"15min":1.512},"last":0.371}]},{"osd":1,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32100000000000001,"15min":0.32400000000000001},"min":{"1min":0.25,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.41799999999999998,"5min":0.47699999999999998,"15min":1.4750000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":3.0710000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309992,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061848,"statfs":{"total":96636764160,"available":96319332352,"internally_reserved":0,"allocated":4415488,"data_stored":2636693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.30499999999999999,"15min":0.30299999999999999},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":0.439,"5min":0.51600000000000001,"15min":1.0229999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34699999999999998},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":1.2949999999999999},"last":0.314}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":3.6360000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42399999999999999,"5min":0.50800000000000001,"15min":1.9099999999999999},"last":0.28999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305828,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066012,"statfs":{"total":96636764160,"available":96323596288,"internally_reserved":0,"allocated":4411392,"data_stored":2635932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.32000000000000001},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.16},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":2.27},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32900000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":2.9169999999999998},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.184},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.4449999999999998},"last":0.374},{"interface":"front","average":{"1min":0.318,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.20399999999999999},"max":{"1min":0.439,"5min":0.49299999999999999,"15min":3.0760000000000001},"last":0.40799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:33.797 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.797 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.797 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.797 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.798 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.798 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.798 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.798 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T11:15:14.288946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.798 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T11:15:09.219587+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=15, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=53, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:53:33.798 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:53:53.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:53:54.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-09-16T11:53:54.087 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-09-16T11:53:54.099 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-09-16T11:53:53.601423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":929364,"kb_used_data":12348,"kb_used_omap":4,"kb_used_meta":916987,"kb_avail":282186156,"statfs":{"total":289910292480,"available":288958623744,"internally_reserved":0,"allocated":12644352,"data_stored":7318931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":938994696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002394"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":475,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:48.863905+0000","last_change":"2024-09-16T11:53:48.863876+0000","last_active":"2024-09-16T11:53:48.863905+0000","last_peered":"2024-09-16T11:53:48.863905+0000","last_clean":"2024-09-16T11:53:48.863905+0000","last_became_active":"2024-09-16T11:15:15.301556+0000","last_became_peered":"2024-09-16T11:15:15.301556+0000","last_unstale":"2024-09-16T11:53:48.863905+0000","last_undegraded":"2024-09-16T11:53:48.863905+0000","last_fullsized":"2024-09-16T11:53:48.863905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:48.863769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:48.863769+0000","last_clean_scrub_stamp":"2024-09-16T11:53:48.863769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:34:29.875416+0000","scrub_duration":0.001442671,"objects_trimmed":0,"snaptrim_duration":0.00068756600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":475,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:42.842159+0000","last_change":"2024-09-16T11:53:42.842128+0000","last_active":"2024-09-16T11:53:42.842159+0000","last_peered":"2024-09-16T11:53:42.842159+0000","last_clean":"2024-09-16T11:53:42.842159+0000","last_became_active":"2024-09-16T11:15:15.299499+0000","last_became_peered":"2024-09-16T11:15:15.299499+0000","last_unstale":"2024-09-16T11:53:42.842159+0000","last_undegraded":"2024-09-16T11:53:42.842159+0000","last_fullsized":"2024-09-16T11:53:42.842159+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:42.842021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:42.842021+0000","last_clean_scrub_stamp":"2024-09-16T11:53:42.842021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:09:03.524081+0000","scrub_duration":0.0013943359999999999,"objects_trimmed":0,"snaptrim_duration":0.00059021699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":475,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:46.853212+0000","last_change":"2024-09-16T11:53:46.853158+0000","last_active":"2024-09-16T11:53:46.853212+0000","last_peered":"2024-09-16T11:53:46.853212+0000","last_clean":"2024-09-16T11:53:46.853212+0000","last_became_active":"2024-09-16T11:15:15.299733+0000","last_became_peered":"2024-09-16T11:15:15.299733+0000","last_unstale":"2024-09-16T11:53:46.853212+0000","last_undegraded":"2024-09-16T11:53:46.853212+0000","last_fullsized":"2024-09-16T11:53:46.853212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:46.853026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:46.853026+0000","last_clean_scrub_stamp":"2024-09-16T11:53:46.853026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:44.897309+0000","scrub_duration":0.0016184299999999999,"objects_trimmed":0,"snaptrim_duration":0.000578113,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":475,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:41.818505+0000","last_change":"2024-09-16T11:53:41.818449+0000","last_active":"2024-09-16T11:53:41.818505+0000","last_peered":"2024-09-16T11:53:41.818505+0000","last_clean":"2024-09-16T11:53:41.818505+0000","last_became_active":"2024-09-16T11:15:15.299412+0000","last_became_peered":"2024-09-16T11:15:15.299412+0000","last_unstale":"2024-09-16T11:53:41.818505+0000","last_undegraded":"2024-09-16T11:53:41.818505+0000","last_fullsized":"2024-09-16T11:53:41.818505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:41.818334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:41.818334+0000","last_clean_scrub_stamp":"2024-09-16T11:53:41.818334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:54:02.607328+0000","scrub_duration":0.0017117670000000001,"objects_trimmed":0,"snaptrim_duration":0.00087536799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"464'3","reported_seq":475,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:32.799531+0000","last_change":"2024-09-16T11:53:32.799489+0000","last_active":"2024-09-16T11:53:32.799531+0000","last_peered":"2024-09-16T11:53:32.799531+0000","last_clean":"2024-09-16T11:53:32.799531+0000","last_became_active":"2024-09-16T11:15:15.299894+0000","last_became_peered":"2024-09-16T11:15:15.299894+0000","last_unstale":"2024-09-16T11:53:32.799531+0000","last_undegraded":"2024-09-16T11:53:32.799531+0000","last_fullsized":"2024-09-16T11:53:32.799531+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'3","last_scrub_stamp":"2024-09-16T11:53:32.799408+0000","last_deep_scrub":"464'3","last_deep_scrub_stamp":"2024-09-16T11:53:32.799408+0000","last_clean_scrub_stamp":"2024-09-16T11:53:32.799408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:59.591043+0000","scrub_duration":0.0021999849999999998,"objects_trimmed":0,"snaptrim_duration":0.00089505400000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":474,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:37.210225+0000","last_change":"2024-09-16T11:53:37.210195+0000","last_active":"2024-09-16T11:53:37.210225+0000","last_peered":"2024-09-16T11:53:37.210225+0000","last_clean":"2024-09-16T11:53:37.210225+0000","last_became_active":"2024-09-16T11:15:15.299940+0000","last_became_peered":"2024-09-16T11:15:15.299940+0000","last_unstale":"2024-09-16T11:53:37.210225+0000","last_undegraded":"2024-09-16T11:53:37.210225+0000","last_fullsized":"2024-09-16T11:53:37.210225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:37.210081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:37.210081+0000","last_clean_scrub_stamp":"2024-09-16T11:53:37.210081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:15:35.036722+0000","scrub_duration":0.001540299,"objects_trimmed":0,"snaptrim_duration":0.00041023200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:38.244243+0000","last_change":"2024-09-16T11:53:38.244211+0000","last_active":"2024-09-16T11:53:38.244243+0000","last_peered":"2024-09-16T11:53:38.244243+0000","last_clean":"2024-09-16T11:53:38.244243+0000","last_became_active":"2024-09-16T11:15:15.298204+0000","last_became_peered":"2024-09-16T11:15:15.298204+0000","last_unstale":"2024-09-16T11:53:38.244243+0000","last_undegraded":"2024-09-16T11:53:38.244243+0000","last_fullsized":"2024-09-16T11:53:38.244243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:53:38.244111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:53:38.244111+0000","last_clean_scrub_stamp":"2024-09-16T11:53:38.244111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:17:46.254408+0000","scrub_duration":0.001668938,"objects_trimmed":0,"snaptrim_duration":0.00039450299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"464'2","reported_seq":477,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:47.865939+0000","last_change":"2024-09-16T11:53:47.865906+0000","last_active":"2024-09-16T11:53:47.865939+0000","last_peered":"2024-09-16T11:53:47.865939+0000","last_clean":"2024-09-16T11:53:47.865939+0000","last_became_active":"2024-09-16T11:15:15.297846+0000","last_became_peered":"2024-09-16T11:15:15.297846+0000","last_unstale":"2024-09-16T11:53:47.865939+0000","last_undegraded":"2024-09-16T11:53:47.865939+0000","last_fullsized":"2024-09-16T11:53:47.865939+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'2","last_scrub_stamp":"2024-09-16T11:53:47.865809+0000","last_deep_scrub":"464'2","last_deep_scrub_stamp":"2024-09-16T11:53:47.865809+0000","last_clean_scrub_stamp":"2024-09-16T11:53:47.865809+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:07.469495+0000","scrub_duration":0.0027989590000000002,"objects_trimmed":0,"snaptrim_duration":0.00072179000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"464'54","reported_seq":541,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-09-16T11:53:38.782121+0000","last_change":"2024-09-16T11:53:38.782069+0000","last_active":"2024-09-16T11:53:38.782121+0000","last_peered":"2024-09-16T11:53:38.782121+0000","last_clean":"2024-09-16T11:53:38.782121+0000","last_became_active":"2024-09-16T11:15:10.225828+0000","last_became_peered":"2024-09-16T11:15:10.225828+0000","last_unstale":"2024-09-16T11:53:38.782121+0000","last_undegraded":"2024-09-16T11:53:38.782121+0000","last_fullsized":"2024-09-16T11:53:38.782121+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'54","last_scrub_stamp":"2024-09-16T11:53:38.781973+0000","last_deep_scrub":"464'54","last_deep_scrub_stamp":"2024-09-16T11:53:38.781973+0000","last_clean_scrub_stamp":"2024-09-16T11:53:38.781973+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:15:06.475523+0000","scrub_duration":0.004481484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313560,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058280,"statfs":{"total":96636764160,"available":96315678720,"internally_reserved":0,"allocated":3817472,"data_stored":2046306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.315,"15min":0.32300000000000001},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.16},"max":{"1min":0.42499999999999999,"5min":0.46300000000000002,"15min":1.482},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.372,"15min":0.376},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":0.51600000000000001,"5min":0.55400000000000005,"15min":1.512},"last":0.245}]},{"osd":1,"last update":"Mon Sep 16 11:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32100000000000001,"15min":0.32400000000000001},"min":{"1min":0.25,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.41799999999999998,"5min":0.47699999999999998,"15min":1.4750000000000001},"last":0.317},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.34399999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":3.0710000000000002},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309984,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061856,"statfs":{"total":96636764160,"available":96319340544,"internally_reserved":0,"allocated":4415488,"data_stored":2636693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.30499999999999999,"15min":0.30299999999999999},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.17499999999999999},"max":{"1min":0.439,"5min":0.51600000000000001,"15min":1.0229999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34699999999999998},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.20200000000000001},"max":{"1min":0.51800000000000002,"5min":0.53200000000000003,"15min":1.2949999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":3.6360000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32800000000000001,"15min":0.34999999999999998},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42399999999999999,"5min":0.50800000000000001,"15min":1.9099999999999999},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305820,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94066020,"statfs":{"total":96636764160,"available":96323604480,"internally_reserved":0,"allocated":4411392,"data_stored":2635932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.32000000000000001},"min":{"1min":0.23899999999999999,"5min":0.187,"15min":0.16},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":2.27},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.32900000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":2.9169999999999998},"last":0.309}]},{"osd":2,"last update":"Mon Sep 16 11:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32400000000000001,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.184},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.4449999999999998},"last":0.37},{"interface":"front","average":{"1min":0.318,"5min":0.32700000000000001,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.20399999999999999},"max":{"1min":0.439,"5min":0.49299999999999999,"15min":3.0760000000000001},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:53:54.100 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-16T11:53:54.388 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T11:53:54.388 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T11:53:54.388 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-16T11:53:54.389 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:54.436 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-16T11:53:54.436 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-16T11:53:54.436 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:54.471 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-16T11:53:54.471 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-16T11:53:54.471 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:54.549 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-16T11:53:54.549 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T11:53:54.549 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-16T11:53:54.550 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:54.613 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-16T11:53:54.613 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T11:53:54.614 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-16T11:53:54.614 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:53:54.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.mon.a is failed for ~0s 2024-09-16T11:53:54.667 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-16T11:53:54.667 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T11:53:54.667 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T11:53:54.734 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:53:54.735 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-16T11:53:54.916 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:53:54.916 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-16T11:53:55.037 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:53:55.038 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-16T11:53:55.096 INFO:tasks.ceph:Archiving mon data... 2024-09-16T11:53:55.096 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/data/cluster1.mon.a.tgz 2024-09-16T11:53:55.096 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-09-16T11:53:55.111 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.sdLHAzLUso 2024-09-16T11:53:55.112 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.sdLHAzLUso 2024-09-16T11:53:55.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.sdLHAzLUso 2024-09-16T11:53:55.372 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.sdLHAzLUso is 383KB 2024-09-16T11:53:55.467 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.sdLHAzLUso 2024-09-16T11:53:55.482 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T11:53:55.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:53:55.525 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:53:55.555 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T11:53:55.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T11:53:55.590 DEBUG:teuthology.orchestra.run.smithi103:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T11:53:55.616 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T11:53:55.628 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T11:53:55.633 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T11:53:55.635 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/remote/smithi001/crash 2024-09-16T11:53:55.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T11:53:55.685 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/remote/smithi103/crash 2024-09-16T11:53:55.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T11:53:55.713 INFO:tasks.ceph:Compressing logs... 2024-09-16T11:53:55.713 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T11:53:55.729 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T11:53:55.751 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32055.log 2024-09-16T11:53:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37434.log 2024-09-16T11:53:55.772 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36223.log 2024-09-16T11:53:55.772 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32055.log.gz 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37350.log 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37434.log.gz 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36223.log.gz 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36075.log 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37350.log.gz 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35677.log 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-mon.a.log: /var/log/ceph/cluster1-client.admin.36075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35601.log 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36075.log.gz 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37392.log 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35677.log.gz 2024-09-16T11:53:55.773 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32015.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35601.log.gz 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35461.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37392.log.gz 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35842.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32015.log.gz 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35461.log.gz 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37476.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35842.log.gz 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36055.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37686.log 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37476.log.gz 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36055.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36055.log.gz -5 2024-09-16T11:53:55.774 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.32154.log 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37266.log 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37686.log.gz 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32154.log.gz 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37518.log 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35719.log 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37266.log.gz 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36265.log 0.0% 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37518.log.gz 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35719.log.gz 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36265.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph.tmp-client.admin.27308.log -- replaced with /var/log/ceph/cluster1-client.admin.36265.log.gz 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-16T11:53:55.775 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.27308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35969.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27308.log.gz 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37770.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37308.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35969.log.gz 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32222.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37770.log.gz 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31975.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37308.log.gz 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.32222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37560.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32222.log.gz 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.31975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36115.log 2024-09-16T11:53:55.776 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31975.log.gz 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37560.log.gz 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37810.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36115.log.gz 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37644.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35626.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37810.log.gz 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37644.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37644.log.gz 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35929.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35626.log: /var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37602.log 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.35626.log.gz 2024-09-16T11:53:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35929.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35929.log.gz 2024-09-16T11:53:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37728.log 2024-09-16T11:53:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37602.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37602.log.gz 2024-09-16T11:53:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-16T11:53:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.37728.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31816.log 2024-09-16T11:53:55.778 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37728.log.gz 2024-09-16T11:53:55.778 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-osd.0.log: 92.5% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-16T11:53:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35545.log 2024-09-16T11:53:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.31816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35968.log 2024-09-16T11:53:55.782 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31816.log.gz 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35761.log 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35545.log.gz 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35968.log.gz 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35841.log 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36181.log 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35761.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35761.log.gz 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.35841.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35843.log 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35841.log.gz 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1-client.admin.36181.log: /var/log/ceph/cluster1-client.admin.35843.log: 0.0% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36181.log.gz 2024-09-16T11:53:55.783 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35843.log.gz 2024-09-16T11:53:55.786 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cluster1.log: 87.1% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-16T11:53:55.788 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72183.log 2024-09-16T11:53:55.788 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53192.log 2024-09-16T11:53:55.789 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.72183.log: /var/log/ceph/cluster1-client.mirror.43879.log 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53899.log 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58888.log 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43879.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53192.log.gz 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72183.log.gz 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65812.log 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43879.log.gz 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53899.log.gz 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67093.log 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61924.log 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65812.log.gz 2024-09-16T11:53:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49267.log 2024-09-16T11:53:55.799 INFO:teuthology.orchestra.run.smithi103.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58888.log.gz 88.2% 2024-09-16T11:53:55.799 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67093.log.gz 2024-09-16T11:53:55.799 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/cluster1-client.mirror.61924.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53035.log 2024-09-16T11:53:55.799 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49267.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61924.log.gz 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43538.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49267.log.gz 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50757.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53035.log.gz 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48094.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43538.log.gz 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/cluster2-client.mirror.50757.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69537.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75280.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50989.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69537.log.gz 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75280.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75280.log.gz 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54911.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59133.log 2024-09-16T11:53:55.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50989.log: /var/log/ceph/cluster1-client.mirror.54911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65998.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59133.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54911.log.gz 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47645.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53819.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65998.log.gz 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38913.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60870.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59133.log.gz 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38913.log: gzip 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.38913.log.gz 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60983.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40495.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53819.log.gz 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45344.log 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60983.log.gz 2024-09-16T11:53:55.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60102.log 2024-09-16T11:53:55.802 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45902.log 2024-09-16T11:53:55.802 INFO:teuthology.orchestra.run.smithi103.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45344.log.gz 2024-09-16T11:53:55.803 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60102.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60102.log.gz 2024-09-16T11:53:55.804 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73320.log 2024-09-16T11:53:55.804 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40994.log 2024-09-16T11:53:55.804 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73320.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73320.log.gz 2024-09-16T11:53:55.806 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45902.log.gz 2024-09-16T11:53:55.807 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46969.log 2024-09-16T11:53:55.808 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50575.log 2024-09-16T11:53:55.808 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46969.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46969.log.gz 2024-09-16T11:53:55.808 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47507.log 2024-09-16T11:53:55.818 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59470.log 2024-09-16T11:53:55.819 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47507.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47507.log.gz 2024-09-16T11:53:55.828 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56166.log 2024-09-16T11:53:55.829 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59470.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59470.log.gz 2024-09-16T11:53:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54373.log 2024-09-16T11:53:55.844 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56166.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56166.log.gz 2024-09-16T11:53:55.856 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52103.log 2024-09-16T11:53:55.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54373.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54373.log.gz 2024-09-16T11:53:55.871 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57436.log 2024-09-16T11:53:55.880 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51505.log 2024-09-16T11:53:55.887 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57436.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60870.log.gz 2024-09-16T11:53:55.893 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41224.log 2024-09-16T11:53:55.893 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51505.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57436.log.gz 2024-09-16T11:53:55.912 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43835.log 2024-09-16T11:53:55.914 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41224.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41224.log.gz 2024-09-16T11:53:55.920 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68029.log 2024-09-16T11:53:55.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43835.log: 90.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66122.log 2024-09-16T11:53:55.933 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68029.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43835.log.gz 2024-09-16T11:53:55.937 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42203.log 2024-09-16T11:53:55.943 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66122.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68029.log.gz 2024-09-16T11:53:55.946 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56094.log 2024-09-16T11:53:55.947 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42203.log: 73.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42203.log.gz 2024-09-16T11:53:55.951 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66122.log.gz 2024-09-16T11:53:55.955 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55728.log 2024-09-16T11:53:55.969 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56094.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56094.log.gz 2024-09-16T11:53:55.971 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73507.log 2024-09-16T11:53:55.971 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55728.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55728.log.gz 2024-09-16T11:53:55.982 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40410.log 2024-09-16T11:53:55.982 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73507.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73507.log.gz 2024-09-16T11:53:55.994 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37386.log 2024-09-16T11:53:55.995 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40410.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40410.log.gz 2024-09-16T11:53:56.002 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64432.log 2024-09-16T11:53:56.011 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68570.log 2024-09-16T11:53:56.014 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64432.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37386.log.gz 2024-09-16T11:53:56.017 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64432.log.gz 2024-09-16T11:53:56.029 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67114.log 2024-09-16T11:53:56.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68570.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68570.log.gz 2024-09-16T11:53:56.045 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66370.log 2024-09-16T11:53:56.046 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67114.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67114.log.gz 2024-09-16T11:53:56.049 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50575.log.gz 2024-09-16T11:53:56.055 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67341.log 2024-09-16T11:53:56.065 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66370.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48675.log 2024-09-16T11:53:56.069 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67341.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66370.log.gz 2024-09-16T11:53:56.077 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38810.log 2024-09-16T11:53:56.078 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-16T11:53:56.079 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48675.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48675.log.gz 2024-09-16T11:53:56.079 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67341.log.gz 2024-09-16T11:53:56.080 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-16T11:53:56.092 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70572.log 2024-09-16T11:53:56.092 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38810.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38810.log.gz 2024-09-16T11:53:56.103 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72018.log 2024-09-16T11:53:56.103 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70572.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70572.log.gz 2024-09-16T11:53:56.112 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67572.log 2024-09-16T11:53:56.115 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72018.log: 88.2% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48094.log.gz 2024-09-16T11:53:56.115 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72018.log.gz 2024-09-16T11:53:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57902.log 2024-09-16T11:53:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67572.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67572.log.gz 2024-09-16T11:53:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58147.log 2024-09-16T11:53:56.118 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57902.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57902.log.gz 2024-09-16T11:53:56.121 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51250.log 2024-09-16T11:53:56.121 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58147.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58147.log.gz 2024-09-16T11:53:56.121 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51091.log 2024-09-16T11:53:56.121 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51250.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51250.log.gz 2024-09-16T11:53:56.121 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47586.log 2024-09-16T11:53:56.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51091.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51091.log.gz 2024-09-16T11:53:56.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62490.log 2024-09-16T11:53:56.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47586.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47586.log.gz 2024-09-16T11:53:56.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64276.log 2024-09-16T11:53:56.125 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62490.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62490.log.gz 2024-09-16T11:53:56.125 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53216.log 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64276.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64276.log.gz 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68757.log 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53216.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53216.log.gz 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52742.log 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68757.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68757.log.gz 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45272.log 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52742.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52742.log.gz 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62652.log 2024-09-16T11:53:56.134 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45272.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45272.log.gz 2024-09-16T11:53:56.135 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58234.log 2024-09-16T11:53:56.135 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62652.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62652.log.gz 2024-09-16T11:53:56.135 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56942.log 2024-09-16T11:53:56.138 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58234.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58234.log.gz 2024-09-16T11:53:56.138 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73276.log 2024-09-16T11:53:56.138 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56942.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56942.log.gz 2024-09-16T11:53:56.139 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68505.log 2024-09-16T11:53:56.139 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73276.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73276.log.gz 2024-09-16T11:53:56.139 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56556.log 2024-09-16T11:53:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68505.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68505.log.gz 2024-09-16T11:53:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.40607.log 2024-09-16T11:53:56.142 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56556.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56556.log.gz 2024-09-16T11:53:56.142 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69042.log 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.40607.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.40607.log.gz 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37682.log 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40495.log.gz 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68277.log 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37682.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69042.log.gz 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37682.log.gz 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69762.log 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32127.log 2024-09-16T11:53:56.146 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68277.log: /var/log/ceph/cluster1-client.mirror.69762.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69762.log.gz 2024-09-16T11:53:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60477.log 2024-09-16T11:53:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.32127.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32127.log.gz 2024-09-16T11:53:56.149 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62238.log 2024-09-16T11:53:56.149 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60477.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68277.log.gz 2024-09-16T11:53:56.149 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72584.log 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62238.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60477.log.gz 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64865.log 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72584.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62238.log.gz 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72584.log.gz 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62770.log 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70730.log 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64865.log.gz 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41896.log 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70730.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70730.log.gz 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65496.log 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41896.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47645.log.gz 2024-09-16T11:53:56.155 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62770.log.gz 2024-09-16T11:53:56.156 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40930.log 2024-09-16T11:53:56.156 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.65496.log: -- /var/log/ceph/cluster1-client.mirror.63284.log 2024-09-16T11:53:56.157 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40930.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65496.log.gz 2024-09-16T11:53:56.157 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65174.log 2024-09-16T11:53:56.159 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63284.log: 92.3% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41896.log.gz 2024-09-16T11:53:56.159 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63284.log.gz 2024-09-16T11:53:56.159 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51113.log 2024-09-16T11:53:56.159 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68611.log 2024-09-16T11:53:56.163 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65174.log: /var/log/ceph/cluster2-client.mirror.51113.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65174.log.gz 2024-09-16T11:53:56.163 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69155.log 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5%/var/log/ceph/cluster2-client.mirror.68611.log: -- replaced with /var/log/ceph/cluster2-client.mirror.40930.log.gz 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47048.log 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69155.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68611.log.gz 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45988.log 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69155.log.gz 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53682.log 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45988.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47048.log.gz 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43927.log 2024-09-16T11:53:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53682.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53682.log.gz 2024-09-16T11:53:56.165 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45296.log 2024-09-16T11:53:56.165 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43927.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43927.log.gz 2024-09-16T11:53:56.166 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37976.log 2024-09-16T11:53:56.166 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45296.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45296.log.gz 2024-09-16T11:53:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47112.log 2024-09-16T11:53:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37976.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37976.log.gz 2024-09-16T11:53:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41598.log 2024-09-16T11:53:56.168 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47112.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45988.log.gz 2024-09-16T11:53:56.168 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47112.log.gz 2024-09-16T11:53:56.169 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68819.log 2024-09-16T11:53:56.169 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56335.log 2024-09-16T11:53:56.169 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68819.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68819.log.gz 2024-09-16T11:53:56.170 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45644.log 2024-09-16T11:53:56.170 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56335.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56335.log.gz 2024-09-16T11:53:56.171 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54976.log 2024-09-16T11:53:56.176 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45644.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45644.log.gz 2024-09-16T11:53:56.177 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60564.log 2024-09-16T11:53:56.178 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54976.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54976.log.gz 2024-09-16T11:53:56.179 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38844.log 2024-09-16T11:53:56.179 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60564.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60564.log.gz 2024-09-16T11:53:56.180 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58212.log 2024-09-16T11:53:56.181 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38844.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38844.log.gz 2024-09-16T11:53:56.181 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67634.log 2024-09-16T11:53:56.183 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58212.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58212.log.gz 2024-09-16T11:53:56.183 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57342.log 2024-09-16T11:53:56.184 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67634.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67634.log.gz 2024-09-16T11:53:56.185 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41031.log 2024-09-16T11:53:56.185 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57342.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57342.log.gz 2024-09-16T11:53:56.186 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40607.log 2024-09-16T11:53:56.187 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.1.41031.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50757.log.gz 2024-09-16T11:53:56.187 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71135.log 2024-09-16T11:53:56.188 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40607.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40607.log.gz 2024-09-16T11:53:56.188 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73572.log 2024-09-16T11:53:56.189 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71135.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71135.log.gz 2024-09-16T11:53:56.189 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58055.log 2024-09-16T11:53:56.191 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73572.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73572.log.gz 2024-09-16T11:53:56.191 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65874.log 2024-09-16T11:53:56.192 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58055.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58055.log.gz 2024-09-16T11:53:56.194 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41428.log 2024-09-16T11:53:56.194 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41031.log.gz 2024-09-16T11:53:56.195 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65874.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65874.log.gz 2024-09-16T11:53:56.195 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48072.log 2024-09-16T11:53:56.195 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44081.log 2024-09-16T11:53:56.197 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48072.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48072.log.gz 2024-09-16T11:53:56.197 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72162.log 2024-09-16T11:53:56.197 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44081.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44081.log.gz 2024-09-16T11:53:56.198 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68865.log 2024-09-16T11:53:56.200 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72162.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72162.log.gz 2024-09-16T11:53:56.200 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39921.log 2024-09-16T11:53:56.200 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68865.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68865.log.gz 2024-09-16T11:53:56.200 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41428.log.gz 2024-09-16T11:53:56.200 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63749.log 2024-09-16T11:53:56.200 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cluster1-client.admin.39921.log: --verbose -- /var/log/ceph/cluster2-client.mirror.48260.log 2024-09-16T11:53:56.202 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63749.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39921.log.gz 2024-09-16T11:53:56.202 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63749.log.gz 2024-09-16T11:53:56.202 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66783.log 2024-09-16T11:53:56.202 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60829.log 2024-09-16T11:53:56.203 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66783.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66783.log.gz 2024-09-16T11:53:56.203 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68525.log 2024-09-16T11:53:56.203 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60829.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60829.log.gz 2024-09-16T11:53:56.204 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61972.log 2024-09-16T11:53:56.206 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68525.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68525.log.gz 2024-09-16T11:53:56.206 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61575.log 2024-09-16T11:53:56.207 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61972.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61972.log.gz 2024-09-16T11:53:56.207 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73190.log 2024-09-16T11:53:56.209 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61575.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61575.log.gz 2024-09-16T11:53:56.210 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58272.log 2024-09-16T11:53:56.230 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48534.log 2024-09-16T11:53:56.243 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58775.log 2024-09-16T11:53:56.245 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48534.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48534.log.gz 2024-09-16T11:53:56.252 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54887.log 2024-09-16T11:53:56.264 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58775.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58775.log.gz 2024-09-16T11:53:56.265 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41174.log 2024-09-16T11:53:56.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54887.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54887.log.gz 2024-09-16T11:53:56.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64610.log 2024-09-16T11:53:56.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41174.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41174.log.gz 2024-09-16T11:53:56.284 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50989.log.gz 2024-09-16T11:53:56.285 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58272.log.gz 2024-09-16T11:53:56.286 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37134.log 2024-09-16T11:53:56.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64610.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64610.log.gz 2024-09-16T11:53:56.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61458.log 2024-09-16T11:53:56.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37134.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37134.log.gz 2024-09-16T11:53:56.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44059.log 2024-09-16T11:53:56.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61458.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61458.log.gz 2024-09-16T11:53:56.289 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44579.log 2024-09-16T11:53:56.294 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44059.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44059.log.gz 2024-09-16T11:53:56.296 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38981.log 2024-09-16T11:53:56.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44579.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44579.log.gz 2024-09-16T11:53:56.298 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39049.log 2024-09-16T11:53:56.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38981.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38981.log.gz 2024-09-16T11:53:56.298 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66597.log 2024-09-16T11:53:56.298 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39049.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39049.log.gz 2024-09-16T11:53:56.300 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71299.log 2024-09-16T11:53:56.300 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66597.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66597.log.gz 2024-09-16T11:53:56.300 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65126.log 2024-09-16T11:53:56.301 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71299.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71299.log.gz 2024-09-16T11:53:56.302 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54224.log 2024-09-16T11:53:56.303 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65126.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65126.log.gz 2024-09-16T11:53:56.303 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42557.log 2024-09-16T11:53:56.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54224.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54224.log.gz 2024-09-16T11:53:56.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60684.log 2024-09-16T11:53:56.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42557.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42557.log.gz 2024-09-16T11:53:56.310 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57103.log 2024-09-16T11:53:56.311 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60684.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60684.log.gz 2024-09-16T11:53:56.312 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69695.log 2024-09-16T11:53:56.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57103.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57103.log.gz 2024-09-16T11:53:56.314 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42719.log 2024-09-16T11:53:56.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69695.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69695.log.gz 2024-09-16T11:53:56.316 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66866.log 2024-09-16T11:53:56.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.42719.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42719.log.gz 2024-09-16T11:53:56.317 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71737.log 2024-09-16T11:53:56.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66866.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66866.log.gz 2024-09-16T11:53:56.318 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73382.log 2024-09-16T11:53:56.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71737.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71737.log.gz 2024-09-16T11:53:56.320 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52123.log 2024-09-16T11:53:56.320 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8%/var/log/ceph/cluster2-client.mirror.73382.log: -- replaced with /var/log/ceph/cluster2-client.mirror.40994.log.gz 2024-09-16T11:53:56.320 INFO:teuthology.orchestra.run.smithi103.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73382.log.gz 2024-09-16T11:53:56.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50258.log 2024-09-16T11:53:56.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.64473.log 2024-09-16T11:53:56.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52123.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52123.log.gz 2024-09-16T11:53:56.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57950.log 2024-09-16T11:53:56.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.0.64473.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50258.log.gz 2024-09-16T11:53:56.337 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63400.log 2024-09-16T11:53:56.338 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57950.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57950.log.gz 2024-09-16T11:53:56.342 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.64473.log.gz 2024-09-16T11:53:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60636.log 2024-09-16T11:53:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62017.log 2024-09-16T11:53:56.344 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63400.log: /var/log/ceph/cluster1-client.mirror.60636.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63400.log.gz 2024-09-16T11:53:56.344 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60636.log.gz 2024-09-16T11:53:56.344 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60660.log 2024-09-16T11:53:56.345 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62017.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62017.log.gz 2024-09-16T11:53:56.346 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57878.log 2024-09-16T11:53:56.346 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63862.log 2024-09-16T11:53:56.347 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60660.log: /var/log/ceph/cluster1-client.mirror.57878.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60660.log.gz 2024-09-16T11:53:56.347 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57878.log.gzgzip 2024-09-16T11:53:56.347 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55749.log 2024-09-16T11:53:56.348 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63862.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63862.log.gz 2024-09-16T11:53:56.349 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42935.log 2024-09-16T11:53:56.349 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47445.log 2024-09-16T11:53:56.349 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55749.log: /var/log/ceph/cluster1-client.mirror.42935.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42935.log.gz 2024-09-16T11:53:56.350 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55749.log.gz 2024-09-16T11:53:56.351 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65688.log 2024-09-16T11:53:56.351 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68463.log 2024-09-16T11:53:56.352 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47445.log: /var/log/ceph/cluster2-client.mirror.65688.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65688.log.gz 2024-09-16T11:53:56.352 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47445.log.gz 2024-09-16T11:53:56.352 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63629.log 2024-09-16T11:53:56.355 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.68463.log: -- /var/log/ceph/cluster1-client.mirror.64651.log 2024-09-16T11:53:56.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63629.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63629.log.gz 2024-09-16T11:53:56.355 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63472.log 2024-09-16T11:53:56.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64651.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68463.log.gz 2024-09-16T11:53:56.356 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46160.log 2024-09-16T11:53:56.356 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63472.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63472.log.gz 2024-09-16T11:53:56.357 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38185.log 2024-09-16T11:53:56.357 INFO:teuthology.orchestra.run.smithi103.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64651.log.gz 2024-09-16T11:53:56.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38060.log 2024-09-16T11:53:56.358 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38185.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38185.log.gz 2024-09-16T11:53:56.358 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66184.log 2024-09-16T11:53:56.358 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38060.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38060.log.gz 2024-09-16T11:53:56.358 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74330.log 2024-09-16T11:53:56.359 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66184.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66184.log.gz 2024-09-16T11:53:56.359 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71000.log 2024-09-16T11:53:56.362 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74330.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46160.log.gz 2024-09-16T11:53:56.363 INFO:teuthology.orchestra.run.smithi103.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74330.log.gz 2024-09-16T11:53:56.363 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57062.log 2024-09-16T11:53:56.364 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51446.log 2024-09-16T11:53:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71000.log: /var/log/ceph/cluster1-client.mirror.57062.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71000.log.gz 2024-09-16T11:53:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52061.log 2024-09-16T11:53:56.365 INFO:teuthology.orchestra.run.smithi103.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57062.log.gz 2024-09-16T11:53:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47349.log 2024-09-16T11:53:56.365 INFO:teuthology.orchestra.run.smithi103.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51446.log.gz 2024-09-16T11:53:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52061.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52061.log.gz 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73551.log 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71507.log 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47349.log.gz 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74904.log 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41598.log.gz/var/log/ceph/cluster2-client.mirror.71507.log: 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73551.log.gz 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65102.log 2024-09-16T11:53:56.370 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45793.log 2024-09-16T11:53:56.371 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74904.log.gz 2024-09-16T11:53:56.371 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52140.log 2024-09-16T11:53:56.371 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45793.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65102.log.gz 2024-09-16T11:53:56.371 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60296.log 2024-09-16T11:53:56.371 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52140.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52140.log.gz 2024-09-16T11:53:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48297.log 2024-09-16T11:53:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60296.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60296.log.gz 2024-09-16T11:53:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66638.log 2024-09-16T11:53:56.373 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48297.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48297.log.gz 2024-09-16T11:53:56.378 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46094.log 2024-09-16T11:53:56.378 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66638.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45793.log.gz 2024-09-16T11:53:56.378 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47242.log 2024-09-16T11:53:56.378 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46094.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66638.log.gz 2024-09-16T11:53:56.378 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70010.log 2024-09-16T11:53:56.382 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47242.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46094.log.gz 2024-09-16T11:53:56.382 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48131.log 2024-09-16T11:53:56.382 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70010.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70010.log.gz 2024-09-16T11:53:56.382 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51505.log.gz 2024-09-16T11:53:56.382 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51170.log 2024-09-16T11:53:56.383 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64586.log 2024-09-16T11:53:56.385 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51170.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51170.log.gz 2024-09-16T11:53:56.385 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48131.log.gz 2024-09-16T11:53:56.385 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65010.log 2024-09-16T11:53:56.385 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.64586.log: -- /var/log/ceph/cluster2-client.mirror.3.73041.log 2024-09-16T11:53:56.386 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64586.log.gz 2024-09-16T11:53:56.386 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63770.log 2024-09-16T11:53:56.386 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.3.73041.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65010.log.gz 2024-09-16T11:53:56.387 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50553.log 2024-09-16T11:53:56.390 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63770.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.73041.log.gz 2024-09-16T11:53:56.390 INFO:teuthology.orchestra.run.smithi103.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63770.log.gz 2024-09-16T11:53:56.391 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64043.log 2024-09-16T11:53:56.391 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49484.log 2024-09-16T11:53:56.391 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50553.log.gz 2024-09-16T11:53:56.391 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68907.log 2024-09-16T11:53:56.391 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49484.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64043.log.gz 2024-09-16T11:53:56.392 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48915.log 2024-09-16T11:53:56.392 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68907.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68907.log.gz 2024-09-16T11:53:56.393 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71253.log 2024-09-16T11:53:56.393 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48915.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48915.log.gz 2024-09-16T11:53:56.394 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57725.log 2024-09-16T11:53:56.406 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67258.log 2024-09-16T11:53:56.407 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57725.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57725.log.gz 2024-09-16T11:53:56.413 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71067.log 2024-09-16T11:53:56.425 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59044.log 2024-09-16T11:53:56.426 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71067.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71067.log.gz 2024-09-16T11:53:56.427 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67258.log.gz 2024-09-16T11:53:56.437 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48359.log 2024-09-16T11:53:56.437 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59044.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59044.log.gz 2024-09-16T11:53:56.443 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52103.log.gz 2024-09-16T11:53:56.445 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71231.log 2024-09-16T11:53:56.445 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48359.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48359.log.gz 2024-09-16T11:53:56.445 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49217.log 2024-09-16T11:53:56.445 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71231.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71231.log.gz 2024-09-16T11:53:56.447 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67134.log 2024-09-16T11:53:56.447 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49217.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49217.log.gz 2024-09-16T11:53:56.447 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46223.log 2024-09-16T11:53:56.450 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67134.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67134.log.gz 2024-09-16T11:53:56.450 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72667.log 2024-09-16T11:53:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46223.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46223.log.gz 2024-09-16T11:53:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38827.log 2024-09-16T11:53:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72667.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72667.log.gz 2024-09-16T11:53:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63043.log 2024-09-16T11:53:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38827.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38827.log.gz 2024-09-16T11:53:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52261.log 2024-09-16T11:53:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63043.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63043.log.gz 2024-09-16T11:53:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47682.log 2024-09-16T11:53:56.473 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72704.log 2024-09-16T11:53:56.474 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47682.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47682.log.gz 2024-09-16T11:53:56.488 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71700.log 2024-09-16T11:53:56.489 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72704.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72704.log.gz 2024-09-16T11:53:56.495 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47028.log 2024-09-16T11:53:56.507 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71700.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71700.log.gz 2024-09-16T11:53:56.508 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37385.log 2024-09-16T11:53:56.509 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47028.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47028.log.gz 2024-09-16T11:53:56.518 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44663.log 2024-09-16T11:53:56.518 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37385.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37385.log.gz 2024-09-16T11:53:56.538 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51272.log 2024-09-16T11:53:56.544 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44663.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44663.log.gz 2024-09-16T11:53:56.547 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68112.log 2024-09-16T11:53:56.548 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51272.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73190.log.gz 2024-09-16T11:53:56.550 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66514.log 2024-09-16T11:53:56.550 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68112.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68112.log.gz 2024-09-16T11:53:56.551 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41939.log 2024-09-16T11:53:56.553 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66514.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66514.log.gz 2024-09-16T11:53:56.553 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72561.log 2024-09-16T11:53:56.558 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41939.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41939.log.gz 2024-09-16T11:53:56.562 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37892.log 2024-09-16T11:53:56.562 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72561.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72561.log.gz 2024-09-16T11:53:56.562 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51113.log.gz 2024-09-16T11:53:56.562 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43975.log 2024-09-16T11:53:56.563 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.37892.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37892.log.gz 2024-09-16T11:53:56.563 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45429.log 2024-09-16T11:53:56.563 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43975.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43975.log.gz 2024-09-16T11:53:56.563 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63537.log 2024-09-16T11:53:56.568 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45429.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45429.log.gz 2024-09-16T11:53:56.568 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65516.log 2024-09-16T11:53:56.570 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63537.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63537.log.gz 2024-09-16T11:53:56.570 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40314.log 2024-09-16T11:53:56.573 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65516.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65516.log.gz 2024-09-16T11:53:56.573 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66432.log 2024-09-16T11:53:56.574 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40314.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40314.log.gz 2024-09-16T11:53:56.574 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59654.log 2024-09-16T11:53:56.575 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66432.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66432.log.gz 2024-09-16T11:53:56.575 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54132.log 2024-09-16T11:53:56.577 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59654.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59654.log.gz 2024-09-16T11:53:56.577 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54642.log 2024-09-16T11:53:56.581 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54132.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54132.log.gz 2024-09-16T11:53:56.581 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42346.log 2024-09-16T11:53:56.581 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54642.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54642.log.gz 2024-09-16T11:53:56.581 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43758.log 2024-09-16T11:53:56.583 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42346.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42346.log.gz 2024-09-16T11:53:56.583 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39831.log 2024-09-16T11:53:56.583 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.43758.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43758.log.gz 2024-09-16T11:53:56.585 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43692.log 2024-09-16T11:53:56.585 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.39831.log: 73.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39831.log.gz 2024-09-16T11:53:56.585 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40794.log 2024-09-16T11:53:56.596 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68839.log 2024-09-16T11:53:56.597 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40794.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40794.log.gz 2024-09-16T11:53:56.603 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65667.log 2024-09-16T11:53:56.612 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68839.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48260.log.gz 2024-09-16T11:53:56.613 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41916.log 2024-09-16T11:53:56.614 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65667.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65667.log.gz 2024-09-16T11:53:56.614 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43514.log 2024-09-16T11:53:56.617 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41916.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68839.log.gz 2024-09-16T11:53:56.619 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41916.log.gz 2024-09-16T11:53:56.620 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62101.log 2024-09-16T11:53:56.620 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71253.log.gz 2024-09-16T11:53:56.620 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71932.log 2024-09-16T11:53:56.623 INFO:teuthology.orchestra.run.smithi103.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43514.log.gz 2024-09-16T11:53:56.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56842.log 2024-09-16T11:53:56.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71932.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62101.log.gz 2024-09-16T11:53:56.623 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71932.log.gz 2024-09-16T11:53:56.623 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56424.log 2024-09-16T11:53:56.623 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49686.log 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56424.log: /var/log/ceph/cluster2-client.mirror.56842.log: 92.7% 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56842.log.gz 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56424.log.gz 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69222.log 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57549.log 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49686.log.gz 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56725.log 2024-09-16T11:53:56.627 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57549.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69222.log.gz 2024-09-16T11:53:56.628 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61278.log 2024-09-16T11:53:56.629 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56725.log: 87.5% 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57549.log.gz 2024-09-16T11:53:56.629 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56725.log.gz 2024-09-16T11:53:56.629 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59358.log 2024-09-16T11:53:56.630 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48871.log 2024-09-16T11:53:56.632 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59358.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61278.log.gz 2024-09-16T11:53:56.632 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59358.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40451.log 2024-09-16T11:53:56.632 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.632 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52002.log 2024-09-16T11:53:56.632 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40451.log: /var/log/ceph/cluster2-client.mirror.48871.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48871.log.gz 2024-09-16T11:53:56.632 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37259.log 2024-09-16T11:53:56.634 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52002.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52002.log.gz 2024-09-16T11:53:56.635 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40167.log 2024-09-16T11:53:56.635 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37259.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37259.log.gz 2024-09-16T11:53:56.635 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59744.log 2024-09-16T11:53:56.640 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40167.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40167.log.gz 2024-09-16T11:53:56.640 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69920.log 2024-09-16T11:53:56.640 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40451.log.gz 2024-09-16T11:53:56.640 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49363.log 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69920.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59744.log.gz 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69920.log.gz 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38338.log 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.49363.log: -- /var/log/ceph/cluster2-client.mirror.73634.log 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38338.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38338.log.gz 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49363.log.gz 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64408.log 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48173.log 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73634.log.gz 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56821.log 2024-09-16T11:53:56.641 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48173.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64408.log.gz 2024-09-16T11:53:56.642 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56050.log 2024-09-16T11:53:56.642 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56821.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56821.log.gz 2024-09-16T11:53:56.643 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64135.log 2024-09-16T11:53:56.643 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56050.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56050.log.gz 2024-09-16T11:53:56.643 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56207.log 2024-09-16T11:53:56.644 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64135.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64135.log.gz 2024-09-16T11:53:56.644 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38399.log 2024-09-16T11:53:56.646 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56207.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56207.log.gz 2024-09-16T11:53:56.647 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63701.log 2024-09-16T11:53:56.647 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38399.log: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38399.log.gz 2024-09-16T11:53:56.648 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37220.log 2024-09-16T11:53:56.648 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63701.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63701.log.gz 2024-09-16T11:53:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40338.log 2024-09-16T11:53:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37220.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37220.log.gz 2024-09-16T11:53:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70662.log 2024-09-16T11:53:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40338.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40338.log.gz 2024-09-16T11:53:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38185.log 2024-09-16T11:53:56.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70662.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70662.log.gz 2024-09-16T11:53:56.651 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73148.log 2024-09-16T11:53:56.654 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.38185.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38185.log.gz 2024-09-16T11:53:56.655 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65894.log 2024-09-16T11:53:56.656 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73148.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73148.log.gz 2024-09-16T11:53:56.656 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60725.log 2024-09-16T11:53:56.658 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65894.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65894.log.gz 2024-09-16T11:53:56.659 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59450.log 2024-09-16T11:53:56.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60725.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60725.log.gz 2024-09-16T11:53:56.661 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49118.log 2024-09-16T11:53:56.662 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59450.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59450.log.gz 2024-09-16T11:53:56.663 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70055.log 2024-09-16T11:53:56.664 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49118.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49118.log.gz 2024-09-16T11:53:56.664 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44976.log 2024-09-16T11:53:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70055.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70055.log.gz 2024-09-16T11:53:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71465.log 2024-09-16T11:53:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44976.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44976.log.gz 2024-09-16T11:53:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.38271.log 2024-09-16T11:53:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71465.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71465.log.gz 2024-09-16T11:53:56.668 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69785.log 2024-09-16T11:53:56.679 INFO:teuthology.orchestra.run.smithi001.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-16T11:53:56.687 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.3.38271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66700.log 2024-09-16T11:53:56.688 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69785.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69785.log.gz 2024-09-16T11:53:56.701 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66928.log 2024-09-16T11:53:56.703 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66700.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66700.log.gz 2024-09-16T11:53:56.714 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52261.log.gz 2024-09-16T11:53:56.714 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52400.log 2024-09-16T11:53:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66928.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66928.log.gz 2024-09-16T11:53:56.715 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57255.log 2024-09-16T11:53:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52400.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52400.log.gz 2024-09-16T11:53:56.717 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41121.log 2024-09-16T11:53:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57255.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57255.log.gz 2024-09-16T11:53:56.718 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37132.log 2024-09-16T11:53:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.41121.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41121.log.gz 2024-09-16T11:53:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71211.log 2024-09-16T11:53:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37132.log.gz 2024-09-16T11:53:56.720 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50026.log 2024-09-16T11:53:56.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71211.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71211.log.gz 2024-09-16T11:53:56.722 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58555.log 2024-09-16T11:53:56.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50026.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50026.log.gz 2024-09-16T11:53:56.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56966.log 2024-09-16T11:53:56.726 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58555.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58555.log.gz 2024-09-16T11:53:56.726 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70820.log 2024-09-16T11:53:56.728 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56966.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56966.log.gz 2024-09-16T11:53:56.728 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73299.log 2024-09-16T11:53:56.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70820.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70820.log.gz 2024-09-16T11:53:56.730 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42863.log 2024-09-16T11:53:56.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73299.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73299.log.gz 2024-09-16T11:53:56.730 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52379.log 2024-09-16T11:53:56.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42863.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42863.log.gz 2024-09-16T11:53:56.732 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65604.log 2024-09-16T11:53:56.733 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52379.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52379.log.gz 2024-09-16T11:53:56.733 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51814.log 2024-09-16T11:53:56.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65604.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65604.log.gz 2024-09-16T11:53:56.734 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47428.log 2024-09-16T11:53:56.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51814.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51814.log.gz 2024-09-16T11:53:56.736 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40386.log 2024-09-16T11:53:56.737 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47428.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47428.log.gz 2024-09-16T11:53:56.737 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68381.log 2024-09-16T11:53:56.738 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40386.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40386.log.gz 2024-09-16T11:53:56.738 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59591.log 2024-09-16T11:53:56.740 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68381.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68381.log.gz 2024-09-16T11:53:56.740 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61707.log 2024-09-16T11:53:56.741 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59591.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59591.log.gz 2024-09-16T11:53:56.741 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66390.log 2024-09-16T11:53:56.742 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61707.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61707.log.gz 2024-09-16T11:53:56.743 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47803.log 2024-09-16T11:53:56.745 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66390.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66390.log.gz 2024-09-16T11:53:56.745 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60364.log 2024-09-16T11:53:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55656.log 2024-09-16T11:53:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60364.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60364.log.gz 2024-09-16T11:53:56.770 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44774.log 2024-09-16T11:53:56.771 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55656.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55656.log.gz 2024-09-16T11:53:56.771 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57392.log 2024-09-16T11:53:56.778 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.44774.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63023.log 2024-09-16T11:53:56.778 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44774.log.gz 2024-09-16T11:53:56.785 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46432.log 2024-09-16T11:53:56.787 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63023.log: /var/log/ceph/cluster2-client.mirror.57392.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57392.log.gz 2024-09-16T11:53:56.787 INFO:teuthology.orchestra.run.smithi103.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63023.log.gz 2024-09-16T11:53:56.788 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71507.log.gz 2024-09-16T11:53:56.788 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55680.log 2024-09-16T11:53:56.790 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46432.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46432.log.gz 2024-09-16T11:53:56.790 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38776.log 2024-09-16T11:53:56.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55680.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55680.log.gz 2024-09-16T11:53:56.792 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45510.log 2024-09-16T11:53:56.793 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38776.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38776.log.gz 2024-09-16T11:53:56.793 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42577.log 2024-09-16T11:53:56.803 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.0.45510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66576.log 2024-09-16T11:53:56.807 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42577.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43692.log.gz 2024-09-16T11:53:56.808 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42577.log.gz 2024-09-16T11:53:56.809 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71022.log 2024-09-16T11:53:56.811 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66576.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66576.log.gz 2024-09-16T11:53:56.812 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64990.log 2024-09-16T11:53:56.813 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71022.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71022.log.gz 2024-09-16T11:53:56.813 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57854.log 2024-09-16T11:53:56.815 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64990.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64990.log.gz 2024-09-16T11:53:56.816 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51133.log 2024-09-16T11:53:56.816 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49484.log.gz 2024-09-16T11:53:56.816 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57854.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57854.log.gz 2024-09-16T11:53:56.817 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47095.log 2024-09-16T11:53:56.817 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.51133.log: --verbose -- /var/log/ceph/cluster2-client.admin.37052.log 2024-09-16T11:53:56.817 INFO:teuthology.orchestra.run.smithi103.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51133.log.gz 2024-09-16T11:53:56.817 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47095.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47095.log.gz 2024-09-16T11:53:56.819 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64364.log 2024-09-16T11:53:56.819 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37052.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57991.log 2024-09-16T11:53:56.819 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.37052.log.gz 2024-09-16T11:53:56.819 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64364.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64364.log.gz 2024-09-16T11:53:56.855 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60388.log 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52202.log 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60388.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.38271.log.gz 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60388.log.gz 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74245.log 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53923.log 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57991.log.gz/var/log/ceph/cluster2-client.mirror.52202.log: 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52202.log.gz 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73715.log 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr: 86.9%/var/log/ceph/cluster1-client.mirror.53923.log: -- replaced with /var/log/ceph/cluster1-client.mirror.74245.log.gz 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- 87.3% /var/log/ceph/cluster1-client.mirror.46008.log 2024-09-16T11:53:56.856 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53923.log.gz 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65936.log 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73715.log.gz/var/log/ceph/cluster1-client.mirror.46008.log: 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60805.log 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65936.log: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70505.log 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.45510.log.gz 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65936.log.gz 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60805.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60805.log.gz 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45320.log 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72475.log 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70505.log.gz/var/log/ceph/cluster1-client.mirror.45320.log: 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61554.log 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45320.log.gz 2024-09-16T11:53:56.857 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59985.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57014.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61554.log.gz 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59985.log: gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59985.log.gz 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72096.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55333.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.72096.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57014.log.gz 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46008.log.gz 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72096.log.gz 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48554.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74310.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48554.log: gzip 89.8% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.55333.log.gz -- 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/cluster1-client.mirror.72496.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48554.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69065.log 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.858 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74310.log: /var/log/ceph/cluster1-client.mirror.72496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60436.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74310.log.gz 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72496.log.gz/var/log/ceph/cluster1-client.mirror.69065.log: 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69065.log.gz 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60959.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39088.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.60959.log: -- replaced with /var/log/ceph/cluster1-client.mirror.60436.log.gz 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54019.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60959.log.gz 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39088.log: gzip -5 79.7% --verbose -- /var/log/ceph/cluster1-client.mirror.70752.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39088.log.gz 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51071.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70752.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54019.log.gz 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- 86.6% /var/log/ceph/cluster2-client.mirror.58075.log 2024-09-16T11:53:56.859 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70752.log.gz 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62346.log 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59700.log 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51071.log.gz 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58075.log.gz 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44800.log 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62346.log.gz/var/log/ceph/cluster2-client.mirror.59700.log: 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65583.log 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42464.log 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59700.log.gz 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55966.log 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42464.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44800.log.gz 2024-09-16T11:53:56.860 INFO:teuthology.orchestra.run.smithi103.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42464.log.gz 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69267.log 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74528.log 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69267.log: /var/log/ceph/cluster2-client.mirror.55966.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69267.log.gz 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65583.log.gz 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40726.log 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74528.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44507.log 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.74528.log.gz 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55966.log.gz 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75114.log 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40726.log: /var/log/ceph/cluster1-client.mirror.44507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68886.log 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75114.log: 89.2% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75114.log.gz 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44507.log.gz 2024-09-16T11:53:56.861 INFO:teuthology.orchestra.run.smithi103.stderr:gzip 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40726.log.gz 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56446.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68886.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68886.log.gz 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68401.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54598.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74988.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56446.log.gz 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.41485.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74988.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74988.log.gz 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67613.log 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54598.log.gz 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46266.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68401.log.gz 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.41485.log: /var/log/ceph/cluster2-client.mirror.67613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70910.log 2024-09-16T11:53:56.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46266.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67613.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.41485.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48201.log 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.70910.log: /var/log/ceph/cluster2-client.mirror.42956.log 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70910.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43063.log 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr: 79.4%/var/log/ceph/cluster2-client.mirror.42956.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48201.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62954.log 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43063.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46266.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46372.log 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62954.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62954.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40115.log 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46372.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46372.log.gz 2024-09-16T11:53:56.863 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45621.log 2024-09-16T11:53:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40115.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.40115.log.gz 2024-09-16T11:53:56.864 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42956.log.gz 2024-09-16T11:53:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40887.log 2024-09-16T11:53:56.864 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44555.log 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40887.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45621.log.gz 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40887.log.gz 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/cluster2-client.mirror.52081.log 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68195.log 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44555.log.gz 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55921.log 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68195.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52081.log.gz 2024-09-16T11:53:56.873 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47702.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68195.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38102.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47702.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55921.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47702.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44039.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49180.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44039.log: /var/log/ceph/cluster2-client.admin.38102.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38102.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54510.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49180.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49180.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51834.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54510.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54510.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46884.log 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51834.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51834.log.gz 2024-09-16T11:53:56.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42003.log 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46884.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46884.log.gz 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65542.log 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44039.log.gz 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64945.log 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65542.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65542.log.gz 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46180.log 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64945.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64945.log.gz 2024-09-16T11:53:56.875 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44439.log 2024-09-16T11:53:56.877 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46180.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46180.log.gz 2024-09-16T11:53:56.878 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41576.log 2024-09-16T11:53:56.879 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44439.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44439.log.gz 2024-09-16T11:53:56.879 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40842.log 2024-09-16T11:53:56.881 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41576.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41576.log.gz 2024-09-16T11:53:56.881 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55035.log 2024-09-16T11:53:56.882 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40842.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40842.log.gz 2024-09-16T11:53:56.883 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70257.log 2024-09-16T11:53:56.885 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55035.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55035.log.gz 2024-09-16T11:53:56.889 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37511.log 2024-09-16T11:53:56.889 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70257.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70257.log.gz 2024-09-16T11:53:56.889 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62620.log 2024-09-16T11:53:56.889 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37511.log: 93.8% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37511.log.gz 2024-09-16T11:53:56.889 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47242.log.gz 2024-09-16T11:53:56.889 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68632.log 2024-09-16T11:53:56.890 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.62620.log: -- /var/log/ceph/cluster2-client.mirror.60747.log 2024-09-16T11:53:56.890 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62620.log.gz 2024-09-16T11:53:56.890 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63103.log 2024-09-16T11:53:56.890 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60747.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68632.log.gz 2024-09-16T11:53:56.890 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57163.log 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60747.log.gz 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40907.log 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57163.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63103.log.gz 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53529.log 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57163.log.gz 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38930.log 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53529.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53529.log.gz 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43041.log 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38930.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.38930.log.gz 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48280.log 2024-09-16T11:53:56.895 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43041.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43041.log.gz 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69807.log 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48280.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48280.log.gz 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62725.log 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69807.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69807.log.gz 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54465.log 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62725.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62725.log.gz 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66349.log 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54465.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40907.log.gz 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54465.log.gz 2024-09-16T11:53:56.901 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53706.log 2024-09-16T11:53:56.902 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.38400.log 2024-09-16T11:53:56.902 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66349.log: /var/log/ceph/cluster1-client.mirror.53706.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66349.log.gz 2024-09-16T11:53:56.902 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53706.log.gz 2024-09-16T11:53:56.902 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65403.log 2024-09-16T11:53:56.902 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68360.log 2024-09-16T11:53:56.905 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.3.38400.log: /var/log/ceph/cluster1-client.mirror.65403.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65403.log.gz 2024-09-16T11:53:56.905 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75072.log 2024-09-16T11:53:56.905 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68360.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68360.log.gz 2024-09-16T11:53:56.905 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43605.log 2024-09-16T11:53:56.905 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75072.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75072.log.gz 2024-09-16T11:53:56.905 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59941.log 2024-09-16T11:53:56.907 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43605.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.38400.log.gz 2024-09-16T11:53:56.907 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63243.log 2024-09-16T11:53:56.908 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59941.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59941.log.gz 2024-09-16T11:53:56.909 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52600.log 2024-09-16T11:53:56.910 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63243.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43605.log.gz 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63243.log.gz 2024-09-16T11:53:56.910 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.910 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73168.log 2024-09-16T11:53:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.52600.log: -- /var/log/ceph/cluster2-client.mirror.52315.log 2024-09-16T11:53:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73168.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73168.log.gz 2024-09-16T11:53:56.912 INFO:teuthology.orchestra.run.smithi103.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52600.log.gz 2024-09-16T11:53:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48052.log 2024-09-16T11:53:56.912 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66948.log 2024-09-16T11:53:56.914 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48052.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52315.log.gz 2024-09-16T11:53:56.914 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61948.log 2024-09-16T11:53:56.914 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66948.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48052.log.gz 2024-09-16T11:53:56.914 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70482.log 2024-09-16T11:53:56.915 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61948.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61948.log.gz 2024-09-16T11:53:56.915 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39982.log 2024-09-16T11:53:56.916 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70482.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70482.log.gz 2024-09-16T11:53:56.916 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66948.log.gz 2024-09-16T11:53:56.916 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72605.log 2024-09-16T11:53:56.918 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36763.log 2024-09-16T11:53:56.918 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39982.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39982.log.gz 2024-09-16T11:53:56.918 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42999.log 2024-09-16T11:53:56.919 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72605.log.gz 2024-09-16T11:53:56.919 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.36763.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36763.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41292.log 2024-09-16T11:53:56.919 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.919 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66225.log 2024-09-16T11:53:56.919 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41292.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41292.log.gz 2024-09-16T11:53:56.920 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37260.log 2024-09-16T11:53:56.920 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66225.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66225.log.gz 2024-09-16T11:53:56.921 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71976.log 2024-09-16T11:53:56.921 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37260.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37260.log.gz 2024-09-16T11:53:56.921 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47154.log 2024-09-16T11:53:56.922 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71976.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71976.log.gz 2024-09-16T11:53:56.922 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49101.log 2024-09-16T11:53:56.923 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47154.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42999.log.gz 2024-09-16T11:53:56.926 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38166.log 2024-09-16T11:53:56.926 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49101.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49101.log.gz 2024-09-16T11:53:56.926 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63559.log 2024-09-16T11:53:56.926 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38166.log: 82.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38166.log.gz 2024-09-16T11:53:56.926 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45202.log 2024-09-16T11:53:56.927 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63559.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63559.log.gz 2024-09-16T11:53:56.927 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66060.log 2024-09-16T11:53:56.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45202.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45202.log.gz 2024-09-16T11:53:56.929 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38667.log 2024-09-16T11:53:56.930 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66060.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66060.log.gz 2024-09-16T11:53:56.932 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43562.log 2024-09-16T11:53:56.932 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.38667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38667.log.gz 2024-09-16T11:53:56.932 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57300.log 2024-09-16T11:53:56.933 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43562.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43562.log.gz 2024-09-16T11:53:56.933 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48596.log 2024-09-16T11:53:56.934 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57300.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57300.log.gz 2024-09-16T11:53:56.934 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37640.log 2024-09-16T11:53:56.937 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48596.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48596.log.gz 2024-09-16T11:53:56.937 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54666.log 2024-09-16T11:53:56.937 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37640.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37640.log.gz 2024-09-16T11:53:56.937 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56008.log 2024-09-16T11:53:56.938 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54666.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54666.log.gz 2024-09-16T11:53:56.938 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42815.log 2024-09-16T11:53:56.940 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56008.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56008.log.gz 2024-09-16T11:53:56.940 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50875.log 2024-09-16T11:53:56.941 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42815.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42815.log.gz 2024-09-16T11:53:56.941 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38256.log 2024-09-16T11:53:56.943 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50875.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50875.log.gz 2024-09-16T11:53:56.943 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55632.log 2024-09-16T11:53:56.951 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.2.38256.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38256.log.gz 2024-09-16T11:53:56.951 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45386.log 2024-09-16T11:53:56.952 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55632.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55632.log.gz 2024-09-16T11:53:56.952 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60122.log 2024-09-16T11:53:56.957 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45386.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51272.log.gz 2024-09-16T11:53:56.960 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60316.log 2024-09-16T11:53:56.960 INFO:teuthology.orchestra.run.smithi103.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45386.log.gz 2024-09-16T11:53:56.960 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43303.log 2024-09-16T11:53:56.960 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60316.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60316.log.gz 2024-09-16T11:53:56.960 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65832.log 2024-09-16T11:53:56.960 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43303.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60122.log.gz 2024-09-16T11:53:56.961 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70955.log 2024-09-16T11:53:56.964 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65832.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65832.log.gz 2024-09-16T11:53:56.964 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58454.log 2024-09-16T11:53:56.964 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70955.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70955.log.gz 2024-09-16T11:53:56.969 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69402.log 2024-09-16T11:53:56.969 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58454.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43303.log.gz 2024-09-16T11:53:56.969 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58454.log.gz 2024-09-16T11:53:56.969 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69515.log 2024-09-16T11:53:56.969 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52790.log 2024-09-16T11:53:56.969 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69402.log.gz 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58430.log 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52790.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69515.log.gz 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66204.log 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52790.log.gz 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40194.log 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66204.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58430.log.gz 2024-09-16T11:53:56.970 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46798.log 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40194.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40194.log.gz 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58013.log 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46798.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66204.log.gz 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46798.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56990.log 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52831.log 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56990.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58013.log.gz 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56990.log.gz 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46907.log 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39066.log 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46907.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46907.log.gz 2024-09-16T11:53:56.975 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50068.log 2024-09-16T11:53:56.976 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39066.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39066.log.gz 2024-09-16T11:53:56.976 INFO:teuthology.orchestra.run.smithi103.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52831.log.gz 2024-09-16T11:53:56.976 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.39831.log 2024-09-16T11:53:56.976 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71090.log 2024-09-16T11:53:56.979 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50068.log: /var/log/ceph/cluster2-client.mirror.1.39831.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.39831.log.gz 2024-09-16T11:53:56.979 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46818.log 2024-09-16T11:53:56.979 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71090.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71090.log.gz 2024-09-16T11:53:56.981 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44016.log 2024-09-16T11:53:56.981 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46818.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46818.log.gz 2024-09-16T11:53:56.981 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42620.log 2024-09-16T11:53:56.987 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44016.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44016.log.gz 2024-09-16T11:53:56.987 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38643.log 2024-09-16T11:53:56.994 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42620.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42620.log.gz 2024-09-16T11:53:56.994 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50912.log 2024-09-16T11:53:56.994 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.38643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38643.log.gz 2024-09-16T11:53:56.994 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48035.log 2024-09-16T11:53:56.998 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.50912.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50912.log.gz 2024-09-16T11:53:56.998 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54956.log 2024-09-16T11:53:56.999 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48035.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48035.log.gz 2024-09-16T11:53:56.999 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43903.log 2024-09-16T11:53:57.002 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54956.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54956.log.gz 2024-09-16T11:53:57.002 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47070.log 2024-09-16T11:53:57.003 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43903.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43903.log.gz 2024-09-16T11:53:57.003 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-16T11:53:57.012 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32260.log 2024-09-16T11:53:57.018 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70235.log 2024-09-16T11:53:57.026 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.32260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61663.log 2024-09-16T11:53:57.026 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32260.log.gz 2024-09-16T11:53:57.033 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70235.log: 86.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69987.log 2024-09-16T11:53:57.033 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70235.log.gz 2024-09-16T11:53:57.040 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61663.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59020.log 2024-09-16T11:53:57.040 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61663.log.gz 2024-09-16T11:53:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69987.log: 86.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74358.log 2024-09-16T11:53:57.047 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69987.log.gz 2024-09-16T11:53:57.054 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59020.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38166.log 2024-09-16T11:53:57.054 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59020.log.gz 2024-09-16T11:53:57.057 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74358.log: 0.0% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72475.log.gz 2024-09-16T11:53:57.067 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74358.log.gz 2024-09-16T11:53:57.070 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56070.log 2024-09-16T11:53:57.083 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.0.38166.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38166.log.gz 2024-09-16T11:53:57.090 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38861.log 2024-09-16T11:53:57.091 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56070.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56070.log.gz 2024-09-16T11:53:57.099 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54248.log 2024-09-16T11:53:57.109 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40770.log 2024-09-16T11:53:57.113 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38861.log.gz 2024-09-16T11:53:57.118 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55475.log 2024-09-16T11:53:57.119 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40770.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40770.log.gz 2024-09-16T11:53:57.122 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54248.log.gz 2024-09-16T11:53:57.131 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71385.log 2024-09-16T11:53:57.132 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55475.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55475.log.gz 2024-09-16T11:53:57.140 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58751.log 2024-09-16T11:53:57.149 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71385.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71385.log.gz 2024-09-16T11:53:57.150 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47154.log.gz 2024-09-16T11:53:57.150 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38947.log 2024-09-16T11:53:57.150 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59155.log 2024-09-16T11:53:57.152 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38947.log: 66.4% -- replaced with /var/log/ceph/cluster2-client.admin.38947.log.gz 2024-09-16T11:53:57.152 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58751.log.gz 2024-09-16T11:53:57.152 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71617.log 2024-09-16T11:53:57.153 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59155.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59155.log.gz 2024-09-16T11:53:57.153 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48633.log 2024-09-16T11:53:57.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71617.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71617.log.gz 2024-09-16T11:53:57.155 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57834.log 2024-09-16T11:53:57.155 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48633.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48633.log.gz 2024-09-16T11:53:57.156 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61820.log 2024-09-16T11:53:57.158 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57834.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57834.log.gz 2024-09-16T11:53:57.158 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64255.log 2024-09-16T11:53:57.160 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61820.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61820.log.gz 2024-09-16T11:53:57.160 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68715.log 2024-09-16T11:53:57.162 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64255.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64255.log.gz 2024-09-16T11:53:57.162 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60457.log 2024-09-16T11:53:57.165 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68715.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68715.log.gz 2024-09-16T11:53:57.165 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63171.log 2024-09-16T11:53:57.167 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60457.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60457.log.gz 2024-09-16T11:53:57.167 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67238.log 2024-09-16T11:53:57.168 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63171.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63171.log.gz 2024-09-16T11:53:57.168 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74179.log 2024-09-16T11:53:57.170 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67238.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67238.log.gz 2024-09-16T11:53:57.170 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68930.log 2024-09-16T11:53:57.171 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74179.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74179.log.gz 2024-09-16T11:53:57.171 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44837.log 2024-09-16T11:53:57.172 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68930.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68930.log.gz 2024-09-16T11:53:57.173 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72326.log 2024-09-16T11:53:57.176 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44837.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44837.log.gz 2024-09-16T11:53:57.176 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61779.log 2024-09-16T11:53:57.178 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72326.log: 86.1% -- replaced with /var/log/ceph/cluster1-client.mirror.72326.log.gz 2024-09-16T11:53:57.178 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66535.log 2024-09-16T11:53:57.179 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61779.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61779.log.gz 2024-09-16T11:53:57.179 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53148.log 2024-09-16T11:53:57.180 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66535.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66535.log.gz 2024-09-16T11:53:57.180 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51423.log 2024-09-16T11:53:57.182 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53148.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48173.log.gz 2024-09-16T11:53:57.182 INFO:teuthology.orchestra.run.smithi103.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53148.log.gz 2024-09-16T11:53:57.182 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69312.log 2024-09-16T11:53:57.183 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40270.log 2024-09-16T11:53:57.183 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69312.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69312.log.gz 2024-09-16T11:53:57.183 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67404.log 2024-09-16T11:53:57.184 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40270.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40270.log.gz 2024-09-16T11:53:57.185 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52024.log 2024-09-16T11:53:57.186 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67404.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67404.log.gz 2024-09-16T11:53:57.186 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62749.log 2024-09-16T11:53:57.199 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62059.log 2024-09-16T11:53:57.199 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62749.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62749.log.gz 2024-09-16T11:53:57.207 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70032.log 2024-09-16T11:53:57.208 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62059.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62059.log.gz 2024-09-16T11:53:57.219 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59861.log 2024-09-16T11:53:57.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70032.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70032.log.gz 2024-09-16T11:53:57.233 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52766.log 2024-09-16T11:53:57.234 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59861.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59861.log.gz 2024-09-16T11:53:57.242 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52895.log 2024-09-16T11:53:57.242 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52766.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52766.log.gz 2024-09-16T11:53:57.255 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50794.log 2024-09-16T11:53:57.256 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52895.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52895.log.gz 2024-09-16T11:53:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58123.log 2024-09-16T11:53:57.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50794.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50794.log.gz 2024-09-16T11:53:57.283 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47070.log.gz 2024-09-16T11:53:57.287 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51797.log 2024-09-16T11:53:57.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58123.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58123.log.gz 2024-09-16T11:53:57.287 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51364.log 2024-09-16T11:53:57.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51797.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51797.log.gz 2024-09-16T11:53:57.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47465.log 2024-09-16T11:53:57.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51364.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51364.log.gz 2024-09-16T11:53:57.288 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56580.log 2024-09-16T11:53:57.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47465.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47465.log.gz 2024-09-16T11:53:57.291 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61140.log 2024-09-16T11:53:57.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56580.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56580.log.gz 2024-09-16T11:53:57.291 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53351.log 2024-09-16T11:53:57.291 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61140.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61140.log.gz 2024-09-16T11:53:57.291 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67072.log 2024-09-16T11:53:57.292 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53351.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53351.log.gz 2024-09-16T11:53:57.292 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46618.log 2024-09-16T11:53:57.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67072.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67072.log.gz 2024-09-16T11:53:57.295 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45137.log 2024-09-16T11:53:57.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46618.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46618.log.gz 2024-09-16T11:53:57.296 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50858.log 2024-09-16T11:53:57.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45137.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45137.log.gz 2024-09-16T11:53:57.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70707.log 2024-09-16T11:53:57.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50858.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50858.log.gz 2024-09-16T11:53:57.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51621.log 2024-09-16T11:53:57.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70707.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70707.log.gz 2024-09-16T11:53:57.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72118.log 2024-09-16T11:53:57.314 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51621.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47803.log.gz 2024-09-16T11:53:57.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.47840.log 2024-09-16T11:53:57.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71911.log 2024-09-16T11:53:57.335 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.0.47840.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72453.log 2024-09-16T11:53:57.336 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71911.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71911.log.gz 2024-09-16T11:53:57.336 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.47840.log.gz 2024-09-16T11:53:57.349 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42512.log 2024-09-16T11:53:57.349 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72453.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72453.log.gz 2024-09-16T11:53:57.356 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51621.log.gz 2024-09-16T11:53:57.356 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71194.log 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43063.log.gz 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52335.log 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42512.log.gz 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53433.log 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71194.log.gz 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62442.log 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52335.log.gz 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60009.log 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53433.log.gz 2024-09-16T11:53:57.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69560.log 2024-09-16T11:53:57.364 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62442.log.gz 2024-09-16T11:53:57.364 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53098.log 2024-09-16T11:53:57.364 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60009.log.gz 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73419.log 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53098.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69560.log.gz 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44158.log 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8%/var/log/ceph/cluster2-client.mirror.73419.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53098.log.gz 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73419.log.gzgzip 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74433.log 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69380.log 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74433.log: /var/log/ceph/cluster2-client.mirror.44158.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74433.log.gz 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61160.log 2024-09-16T11:53:57.365 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69380.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69380.log.gz 2024-09-16T11:53:57.366 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45117.log 2024-09-16T11:53:57.366 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61160.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61160.log.gz 2024-09-16T11:53:57.366 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54530.log 2024-09-16T11:53:57.367 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45117.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50068.log.gz 2024-09-16T11:53:57.368 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45117.log.gz 2024-09-16T11:53:57.376 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69087.log 2024-09-16T11:53:57.376 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70887.log 2024-09-16T11:53:57.376 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69087.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69087.log.gz 2024-09-16T11:53:57.376 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52239.log 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54530.log.gz 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36892.log 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70887.log.gz/var/log/ceph/cluster2-client.mirror.52239.log: 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61208.log 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52239.log.gz 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.36892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62193.log 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61208.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36892.log.gz 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46839.log 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61208.log.gz 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40950.log 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46839.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62193.log.gz 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46839.log.gz 2024-09-16T11:53:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53550.log 2024-09-16T11:53:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.40950.log: -- /var/log/ceph/cluster1-client.admin.40071.log 2024-09-16T11:53:57.386 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53550.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53550.log.gz 2024-09-16T11:53:57.386 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51893.log 2024-09-16T11:53:57.386 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40071.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40071.log.gz 2024-09-16T11:53:57.386 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63677.log 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51893.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40950.log.gz 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51893.log.gz 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73131.log 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38964.log 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63677.log.gz 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63653.log 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73131.log.gz 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38964.log: 53.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38964.log.gz 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64631.log 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53592.log 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64631.log: /var/log/ceph/cluster1-client.mirror.63653.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63653.log.gz 2024-09-16T11:53:57.387 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74223.log 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53592.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64631.log.gz 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53592.log.gz 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52577.log 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56383.log 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52577.log: 88.6% 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52577.log.gz 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74223.log.gz 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46289.log 2024-09-16T11:53:57.388 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58368.log 2024-09-16T11:53:57.389 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56383.log: /var/log/ceph/cluster2-client.mirror.46289.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56383.log.gz 2024-09-16T11:53:57.389 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49442.log 2024-09-16T11:53:57.389 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58368.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58368.log.gz 2024-09-16T11:53:57.390 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44336.log 2024-09-16T11:53:57.390 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49442.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49442.log.gz 2024-09-16T11:53:57.390 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41827.log 2024-09-16T11:53:57.391 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.44336.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44336.log.gz 2024-09-16T11:53:57.392 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41244.log 2024-09-16T11:53:57.392 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41827.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41827.log.gz 2024-09-16T11:53:57.392 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46289.log.gz 2024-09-16T11:53:57.392 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54441.log 2024-09-16T11:53:57.395 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70932.log 2024-09-16T11:53:57.395 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41244.log: /var/log/ceph/cluster1-client.mirror.54441.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41244.log.gz 2024-09-16T11:53:57.395 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54441.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45879.log 2024-09-16T11:53:57.395 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:57.395 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59092.log 2024-09-16T11:53:57.396 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70932.log: /var/log/ceph/cluster1-client.mirror.45879.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70932.log.gz 2024-09-16T11:53:57.396 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45366.log 2024-09-16T11:53:57.400 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59092.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59092.log.gz 2024-09-16T11:53:57.400 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.67836.log 2024-09-16T11:53:57.402 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45366.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45879.log.gz 2024-09-16T11:53:57.402 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37724.log 2024-09-16T11:53:57.403 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.2.67836.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45366.log.gz 2024-09-16T11:53:57.404 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63954.log 2024-09-16T11:53:57.404 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.37724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37724.log.gz 2024-09-16T11:53:57.405 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45510.log 2024-09-16T11:53:57.405 INFO:teuthology.orchestra.run.smithi103.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.67836.log.gz 2024-09-16T11:53:57.405 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63954.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43951.log 2024-09-16T11:53:57.405 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63954.log.gz 2024-09-16T11:53:57.413 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45922.log 2024-09-16T11:53:57.414 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43951.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43951.log.gz 2024-09-16T11:53:57.415 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44158.log.gz 2024-09-16T11:53:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57788.log 2024-09-16T11:53:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66473.log 2024-09-16T11:53:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45922.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45510.log.gz 2024-09-16T11:53:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67655.log 2024-09-16T11:53:57.418 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66473.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57788.log.gz 2024-09-16T11:53:57.418 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66473.log.gz 2024-09-16T11:53:57.480 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67967.log 2024-09-16T11:53:57.480 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45773.log 2024-09-16T11:53:57.480 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67655.log.gz 2024-09-16T11:53:57.481 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74458.log 2024-09-16T11:53:57.481 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45773.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67967.log.gz 2024-09-16T11:53:57.481 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53658.log 2024-09-16T11:53:57.481 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45922.log.gz 2024-09-16T11:53:57.481 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74458.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65475.log 2024-09-16T11:53:57.481 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74458.log.gz 2024-09-16T11:53:57.482 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42003.log.gz 2024-09-16T11:53:57.482 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45859.log 2024-09-16T11:53:57.482 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53658.log.gz 2024-09-16T11:53:57.482 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56187.log 2024-09-16T11:53:57.482 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45859.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65475.log.gz 2024-09-16T11:53:57.483 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58707.log 2024-09-16T11:53:57.483 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67155.log 2024-09-16T11:53:57.483 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56187.log: /var/log/ceph/cluster2-client.mirror.58707.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45773.log.gz 2024-09-16T11:53:57.483 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58972.log 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58707.log.gz 2024-09-16T11:53:57.483 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71552.log 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67155.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56187.log.gz 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58972.log: gzip 88.0% -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56862.log 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67155.log.gz 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71552.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54489.log 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58972.log.gz 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56862.log: -- replaced with /var/log/ceph/cluster1-client.mirror.71552.log.gz 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57231.log 2024-09-16T11:53:57.484 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44483.log 2024-09-16T11:53:57.485 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.57231.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54489.log.gz 2024-09-16T11:53:57.485 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56862.log.gz 2024-09-16T11:53:57.485 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57231.log.gzgzip 2024-09-16T11:53:57.485 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72519.log 2024-09-16T11:53:57.485 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45859.log.gz 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58192.log 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69290.log 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44483.log.gz 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47566.log 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72519.log.gz 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.69290.log: --verbose -- /var/log/ceph/cluster1-client.mirror.69425.log 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47566.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69290.log.gz 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67176.log 2024-09-16T11:53:57.486 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.62531.log 2024-09-16T11:53:57.487 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69425.log.gz 2024-09-16T11:53:57.487 INFO:teuthology.orchestra.run.smithi103.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58192.log.gz 2024-09-16T11:53:57.487 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46698.log 2024-09-16T11:53:57.487 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67176.log.gz/var/log/ceph/cluster1-client.mirror.2.62531.log: 2024-09-16T11:53:57.487 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51230.log 2024-09-16T11:53:57.487 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42839.log 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.51230.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46698.log.gz 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51230.log.gz 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66990.log 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69830.log 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42839.log.gz 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66990.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66990.log.gz 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/cluster2-client.mirror.68174.log 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58910.log 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69830.log: 86.6%/var/log/ceph/cluster2-client.mirror.68174.log: -- replaced with /var/log/ceph/cluster1-client.mirror.69830.log.gz 2024-09-16T11:53:57.488 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55154.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58910.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68174.log.gz 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64318.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55154.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58910.log.gz 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61072.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64318.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55154.log.gz 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59612.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64318.log.gz 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63002.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59612.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61072.log.gz 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59612.log.gz 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62978.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.63002.log: -- /var/log/ceph/cluster1-client.mirror.57508.log 2024-09-16T11:53:57.489 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63002.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62978.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62978.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65198.log 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45072.log 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57508.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36968.log 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.45072.log: -- replaced with /var/log/ceph/cluster1-client.mirror.65198.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:gzip 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45072.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47761.log 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47524.log 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.36968.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36968.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47761.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47761.log.gz 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40818.log 2024-09-16T11:53:57.490 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53168.log 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40818.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47524.log.gz 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40818.log.gz 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55311.log 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.53168.log: -- /var/log/ceph/cluster2-client.mirror.67031.log 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53168.log.gz 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55311.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55311.log.gz 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70865.log 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54200.log 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1%/var/log/ceph/cluster1-client.mirror.70865.log: -- replaced with /var/log/ceph/cluster2-client.mirror.67031.log.gz 2024-09-16T11:53:57.491 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70865.log.gz 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70122.log 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61257.log 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54200.log.gz 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66845.log 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7%/var/log/ceph/cluster1-client.mirror.61257.log: -- replaced with /var/log/ceph/cluster1-client.mirror.70122.log.gz 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61257.log.gz 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/cluster2-client.mirror.56229.log 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63264.log 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66845.log: /var/log/ceph/cluster2-client.mirror.56229.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66845.log.gz 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56229.log.gz 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57766.log 2024-09-16T11:53:57.492 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57083.log 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57766.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63264.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54082.log 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57083.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57766.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52811.log 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54082.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54082.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57083.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70685.log 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71529.log 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52811.log: /var/log/ceph/cluster1-client.mirror.70685.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70685.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59217.log 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71529.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71529.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52811.log.gz 2024-09-16T11:53:57.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52987.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59217.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59217.log.gz 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49425.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2.audit.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52987.log.gz 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73693.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49425.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49425.log.gz 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70302.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73693.log: 92.2% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65750.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70302.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70302.log.gz 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62145.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65750.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65750.log.gz 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37598.log 2024-09-16T11:53:57.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62145.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62145.log.gz 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72726.log 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37598.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37598.log.gz 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39943.log 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72726.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.62531.log.gz 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44707.log 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.39943.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39943.log.gz 2024-09-16T11:53:57.495 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61052.log 2024-09-16T11:53:57.501 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44707.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51423.log.gz 2024-09-16T11:53:57.501 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66886.log 2024-09-16T11:53:57.501 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61052.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61052.log.gz 2024-09-16T11:53:57.501 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36836.log 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66886.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66886.log.gz 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54863.log 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.36836.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36836.log.gz 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59566.log 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54863.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54863.log.gz 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49239.log 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59566.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59566.log.gz 2024-09-16T11:53:57.502 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73399.log 2024-09-16T11:53:57.504 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69245.log 2024-09-16T11:53:57.509 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73399.log: 89.6% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52024.log.gz 2024-09-16T11:53:57.509 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73399.log.gz 2024-09-16T11:53:57.517 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45664.log 2024-09-16T11:53:57.518 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69245.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69245.log.gz 2024-09-16T11:53:57.519 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52357.log 2024-09-16T11:53:57.521 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45664.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72118.log.gz 2024-09-16T11:53:57.522 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46332.log 2024-09-16T11:53:57.525 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52357.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45664.log.gz 2024-09-16T11:53:57.525 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54819.log 2024-09-16T11:53:57.525 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46332.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46332.log.gz 2024-09-16T11:53:57.526 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47744.log 2024-09-16T11:53:57.527 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54819.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54819.log.gz 2024-09-16T11:53:57.528 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49284.log 2024-09-16T11:53:57.528 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47744.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47744.log.gz 2024-09-16T11:53:57.529 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67196.log 2024-09-16T11:53:57.531 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49284.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49284.log.gz 2024-09-16T11:53:57.532 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38256.log 2024-09-16T11:53:57.533 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67196.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67196.log.gz 2024-09-16T11:53:57.534 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68422.log 2024-09-16T11:53:57.544 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38256.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38256.log.gz 2024-09-16T11:53:57.544 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50009.log 2024-09-16T11:53:57.545 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68422.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68422.log.gz 2024-09-16T11:53:57.545 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46718.log 2024-09-16T11:53:57.545 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50009.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50009.log.gz 2024-09-16T11:53:57.545 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64296.log 2024-09-16T11:53:57.545 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46718.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46718.log.gz 2024-09-16T11:53:57.545 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48814.log 2024-09-16T11:53:57.546 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64296.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64296.log.gz 2024-09-16T11:53:57.546 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42396.log 2024-09-16T11:53:57.564 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42536.log 2024-09-16T11:53:57.565 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42396.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42396.log.gz 2024-09-16T11:53:57.579 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48317.log 2024-09-16T11:53:57.580 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42536.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42536.log.gz 2024-09-16T11:53:57.598 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66804.log 2024-09-16T11:53:57.598 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48317.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48317.log.gz 2024-09-16T11:53:57.605 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54040.log 2024-09-16T11:53:57.605 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66804.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66804.log.gz 2024-09-16T11:53:57.616 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64673.log 2024-09-16T11:53:57.617 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54040.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54040.log.gz 2024-09-16T11:53:57.625 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46051.log 2024-09-16T11:53:57.625 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64673.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64673.log.gz 2024-09-16T11:53:57.634 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61842.log 2024-09-16T11:53:57.639 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46051.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46051.log.gz 2024-09-16T11:53:57.649 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72726.log.gz 2024-09-16T11:53:57.650 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44531.log 2024-09-16T11:53:57.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61842.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61842.log.gz 2024-09-16T11:53:57.651 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56773.log 2024-09-16T11:53:57.652 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44531.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44531.log.gz 2024-09-16T11:53:57.652 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62258.log 2024-09-16T11:53:57.653 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56773.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56773.log.gz 2024-09-16T11:53:57.653 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45707.log 2024-09-16T11:53:57.656 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62258.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62258.log.gz 2024-09-16T11:53:57.656 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71045.log 2024-09-16T11:53:57.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45707.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45707.log.gz 2024-09-16T11:53:57.663 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61320.log 2024-09-16T11:53:57.663 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71045.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71045.log.gz 2024-09-16T11:53:57.663 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50300.log 2024-09-16T11:53:57.664 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61320.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61320.log.gz 2024-09-16T11:53:57.665 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46117.log 2024-09-16T11:53:57.677 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74408.log 2024-09-16T11:53:57.679 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46117.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44707.log.gz 2024-09-16T11:53:57.682 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46117.log.gz 2024-09-16T11:53:57.684 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54735.log 2024-09-16T11:53:57.684 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74408.log.gz 2024-09-16T11:53:57.684 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72877.log 2024-09-16T11:53:57.686 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54735.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54735.log.gz 2024-09-16T11:53:57.686 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60785.log 2024-09-16T11:53:57.687 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72877.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72877.log.gz 2024-09-16T11:53:57.688 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63065.log 2024-09-16T11:53:57.690 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60785.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60785.log.gz 2024-09-16T11:53:57.690 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66080.log 2024-09-16T11:53:57.690 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63065.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63065.log.gz 2024-09-16T11:53:57.691 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48655.log 2024-09-16T11:53:57.694 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66080.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66080.log.gz 2024-09-16T11:53:57.694 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65427.log 2024-09-16T11:53:57.705 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43266.log 2024-09-16T11:53:57.706 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65427.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.65427.log.gz 2024-09-16T11:53:57.719 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71573.log 2024-09-16T11:53:57.721 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43266.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43266.log.gz 2024-09-16T11:53:57.721 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74115.log 2024-09-16T11:53:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71573.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71573.log.gz 2024-09-16T11:53:57.736 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71659.log 2024-09-16T11:53:57.737 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74115.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74115.log.gz 2024-09-16T11:53:57.743 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45965.log 2024-09-16T11:53:57.756 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71659.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71659.log.gz 2024-09-16T11:53:57.757 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52357.log.gz 2024-09-16T11:53:57.757 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58996.log 2024-09-16T11:53:57.759 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45965.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73693.log.gz 2024-09-16T11:53:57.760 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72269.log 2024-09-16T11:53:57.761 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58996.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58996.log.gz 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56508.log 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72269.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72269.log.gz 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73441.log 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56508.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45965.log.gz 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56508.log.gz 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72835.log 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38018.log 2024-09-16T11:53:57.764 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72835.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72835.log.gz 2024-09-16T11:53:57.766 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66018.log 2024-09-16T11:53:57.766 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38018.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38018.log.gz 2024-09-16T11:53:57.766 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52963.log 2024-09-16T11:53:57.769 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66018.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66018.log.gz 2024-09-16T11:53:57.769 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74098.log 2024-09-16T11:53:57.771 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52963.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52963.log.gz 2024-09-16T11:53:57.771 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55612.log 2024-09-16T11:53:57.771 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74098.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74098.log.gz 2024-09-16T11:53:57.771 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38400.log 2024-09-16T11:53:57.776 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55612.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55612.log.gz 2024-09-16T11:53:57.776 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42440.log 2024-09-16T11:53:57.776 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38400.log: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38400.log.gz 2024-09-16T11:53:57.777 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37512.log 2024-09-16T11:53:57.777 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42440.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42440.log.gz 2024-09-16T11:53:57.777 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70212.log 2024-09-16T11:53:57.777 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37512.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37512.log.gz 2024-09-16T11:53:57.777 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47544.log 2024-09-16T11:53:57.779 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70212.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70212.log.gz 2024-09-16T11:53:57.779 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65708.log 2024-09-16T11:53:57.779 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47544.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47544.log.gz 2024-09-16T11:53:57.779 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53638.log 2024-09-16T11:53:57.780 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65708.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49239.log.gz 2024-09-16T11:53:57.781 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43628.log 2024-09-16T11:53:57.782 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53638.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65708.log.gz 2024-09-16T11:53:57.782 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73234.log 2024-09-16T11:53:57.784 INFO:teuthology.orchestra.run.smithi103.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53638.log.gz 2024-09-16T11:53:57.784 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71759.log 2024-09-16T11:53:57.784 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73234.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73234.log.gz 2024-09-16T11:53:57.784 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71797.log 2024-09-16T11:53:57.788 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71759.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43628.log.gz 2024-09-16T11:53:57.788 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38725.log 2024-09-16T11:53:57.788 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71797.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71797.log.gz 2024-09-16T11:53:57.791 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71953.log 2024-09-16T11:53:57.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.38725.log: 67.4% -- replaced with /var/log/ceph/cluster1-client.admin.38725.log.gz 2024-09-16T11:53:57.791 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45730.log 2024-09-16T11:53:57.791 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71953.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71953.log.gz 2024-09-16T11:53:57.791 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58847.log 2024-09-16T11:53:57.798 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45730.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45730.log.gz 2024-09-16T11:53:57.798 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68484.log 2024-09-16T11:53:57.798 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58847.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58847.log.gz 2024-09-16T11:53:57.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.72934.log 2024-09-16T11:53:57.800 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68484.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68484.log.gz 2024-09-16T11:53:57.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62700.log 2024-09-16T11:53:57.804 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.0.72934.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.72934.log.gz 2024-09-16T11:53:57.804 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49745.log 2024-09-16T11:53:57.806 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62700.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62700.log.gz 2024-09-16T11:53:57.806 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45096.log 2024-09-16T11:53:57.820 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64969.log 2024-09-16T11:53:57.821 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45096.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45096.log.gz 2024-09-16T11:53:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46678.log 2024-09-16T11:53:57.834 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64969.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64969.log.gz 2024-09-16T11:53:57.849 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42692.log 2024-09-16T11:53:57.849 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46678.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46678.log.gz 2024-09-16T11:53:57.861 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53879.log 2024-09-16T11:53:57.862 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.42692.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42692.log.gz 2024-09-16T11:53:57.871 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63356.log 2024-09-16T11:53:57.880 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52650.log 2024-09-16T11:53:57.884 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63356.log: 89.9% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53879.log.gz 2024-09-16T11:53:57.889 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72416.log 2024-09-16T11:53:57.889 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63356.log.gz 2024-09-16T11:53:57.889 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52650.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52650.log.gz 2024-09-16T11:53:57.890 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47566.log.gz 2024-09-16T11:53:57.891 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41646.log 2024-09-16T11:53:57.892 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72416.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72416.log.gz 2024-09-16T11:53:57.892 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54714.log 2024-09-16T11:53:57.893 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.41646.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41646.log.gz 2024-09-16T11:53:57.893 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67320.log 2024-09-16T11:53:57.893 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54714.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54714.log.gz 2024-09-16T11:53:57.894 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73951.log 2024-09-16T11:53:57.898 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67320.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67320.log.gz 2024-09-16T11:53:57.898 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66742.log 2024-09-16T11:53:57.898 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73951.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73951.log.gz 2024-09-16T11:53:57.900 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41779.log 2024-09-16T11:53:57.900 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66742.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66742.log.gz 2024-09-16T11:53:57.900 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50516.log 2024-09-16T11:53:57.900 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41779.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41779.log.gz 2024-09-16T11:53:57.903 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70280.log 2024-09-16T11:53:57.903 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50516.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50516.log.gz 2024-09-16T11:53:57.903 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65563.log 2024-09-16T11:53:57.903 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70280.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70280.log.gz 2024-09-16T11:53:57.903 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42664.log 2024-09-16T11:53:57.903 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65563.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65563.log.gz 2024-09-16T11:53:57.904 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47623.log 2024-09-16T11:53:57.915 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56118.log 2024-09-16T11:53:57.915 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47623.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47623.log.gz 2024-09-16T11:53:57.929 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74008.log 2024-09-16T11:53:57.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56118.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56118.log.gz 2024-09-16T11:53:57.943 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68009.log 2024-09-16T11:53:57.943 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74008.log: 74.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74008.log.gz 2024-09-16T11:53:57.952 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43466.log 2024-09-16T11:53:57.958 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68009.log: 88.2% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68009.log.gz 2024-09-16T11:53:57.958 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48814.log.gz 2024-09-16T11:53:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42745.log 2024-09-16T11:53:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43466.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43466.log.gz 2024-09-16T11:53:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.74008.log 2024-09-16T11:53:57.962 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42745.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42745.log.gz 2024-09-16T11:53:57.962 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74383.log 2024-09-16T11:53:57.965 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.0.74008.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.74008.log.gz 2024-09-16T11:53:57.965 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46203.log 2024-09-16T11:53:57.965 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74383.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74383.log.gz 2024-09-16T11:53:57.966 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.47946.log 2024-09-16T11:53:57.972 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46203.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46203.log.gz 2024-09-16T11:53:57.972 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69200.log 2024-09-16T11:53:57.985 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.47946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72433.log 2024-09-16T11:53:57.985 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69200.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69200.log.gz 2024-09-16T11:53:57.998 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69897.log 2024-09-16T11:53:57.999 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72433.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72433.log.gz 2024-09-16T11:53:58.000 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68975.log 2024-09-16T11:53:58.007 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69897.log: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56667.log 2024-09-16T11:53:58.007 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69897.log.gz 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68975.log: 86.7% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48655.log.gz 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68975.log.gz 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50300.log.gz 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48114.log 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60340.log 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48114.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48114.log.gz 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56667.log.gz 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62812.log 2024-09-16T11:53:58.028 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.60340.log: -- /var/log/ceph/cluster2-client.mirror.50656.log 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60340.log.gz 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62676.log 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50656.log: 87.5% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62812.log.gz 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50656.log.gz 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67363.log 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75030.log 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62676.log.gz 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74946.log 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75030.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75030.log.gz 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67363.log.gz 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55174.log 2024-09-16T11:53:58.029 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74946.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51735.log 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.74946.log.gz 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/cluster2-client.mirror.55174.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47270.log 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55174.log.gz 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38708.log 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47270.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47270.log.gz 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68091.log 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51735.log.gz 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.38708.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.38708.log.gz 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.49012.log 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54755.log 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68091.log: /var/log/ceph/cluster1-client.mirror.0.49012.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68091.log.gz 2024-09-16T11:53:58.030 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70190.log 2024-09-16T11:53:58.032 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54755.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54755.log.gz 2024-09-16T11:53:58.032 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44620.log 2024-09-16T11:53:58.032 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70190.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70190.log.gz 2024-09-16T11:53:58.033 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67737.log 2024-09-16T11:53:58.036 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44620.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44620.log.gz 2024-09-16T11:53:58.036 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72626.log 2024-09-16T11:53:58.039 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67737.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67737.log.gz 2024-09-16T11:53:58.039 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55791.log 2024-09-16T11:53:58.040 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72626.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72626.log.gz 2024-09-16T11:53:58.040 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65977.log 2024-09-16T11:53:58.041 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55791.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55791.log.gz 2024-09-16T11:53:58.043 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70167.log 2024-09-16T11:53:58.043 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65977.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65977.log.gz 2024-09-16T11:53:58.043 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66721.log 2024-09-16T11:53:58.045 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70167.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70167.log.gz 2024-09-16T11:53:58.045 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63496.log 2024-09-16T11:53:58.045 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66721.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66721.log.gz 2024-09-16T11:53:58.047 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52718.log 2024-09-16T11:53:58.047 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63496.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63496.log.gz 2024-09-16T11:53:58.047 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48455.log 2024-09-16T11:53:58.049 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52718.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52718.log.gz 2024-09-16T11:53:58.049 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56884.log 2024-09-16T11:53:58.050 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48455.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48455.log.gz 2024-09-16T11:53:58.050 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47329.log 2024-09-16T11:53:58.052 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56884.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56884.log.gz 2024-09-16T11:53:58.052 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44643.log 2024-09-16T11:53:58.066 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47329.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.49012.log.gz 2024-09-16T11:53:58.067 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51054.log 2024-09-16T11:53:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44643.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44643.log.gz 2024-09-16T11:53:58.073 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60542.log 2024-09-16T11:53:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51054.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51054.log.gz 2024-09-16T11:53:58.074 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40473.log 2024-09-16T11:53:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60542.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60542.log.gz 2024-09-16T11:53:58.076 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46859.log 2024-09-16T11:53:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40473.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40473.log.gz 2024-09-16T11:53:58.077 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70460.log 2024-09-16T11:53:58.081 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46859.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46859.log.gz 2024-09-16T11:53:58.081 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51718.log 2024-09-16T11:53:58.081 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70460.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70460.log.gz 2024-09-16T11:53:58.083 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56797.log 2024-09-16T11:53:58.083 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51718.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51718.log.gz 2024-09-16T11:53:58.083 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48376.log 2024-09-16T11:53:58.084 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56797.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56797.log.gz 2024-09-16T11:53:58.085 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38793.log 2024-09-16T11:53:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48376.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48376.log.gz 2024-09-16T11:53:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65263.log 2024-09-16T11:53:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38793.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38793.log.gz 2024-09-16T11:53:58.087 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43855.log 2024-09-16T11:53:58.090 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65263.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65263.log.gz 2024-09-16T11:53:58.090 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62906.log 2024-09-16T11:53:58.091 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43855.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43855.log.gz 2024-09-16T11:53:58.091 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67799.log 2024-09-16T11:53:58.092 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62906.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62906.log.gz 2024-09-16T11:53:58.093 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53409.log 2024-09-16T11:53:58.095 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67799.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67799.log.gz 2024-09-16T11:53:58.095 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41851.log 2024-09-16T11:53:58.097 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53409.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53409.log.gz 2024-09-16T11:53:58.097 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41619.log 2024-09-16T11:53:58.098 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41851.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41851.log.gz 2024-09-16T11:53:58.098 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68153.log 2024-09-16T11:53:58.099 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.41619.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41619.log.gz 2024-09-16T11:53:58.099 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73796.log 2024-09-16T11:53:58.102 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68153.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68153.log.gz 2024-09-16T11:53:58.102 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45048.log 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73796.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73796.log.gz 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49383.log 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45048.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45048.log.gz 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72139.log 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49383.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49383.log.gz 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65150.log 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72139.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72139.log.gz 2024-09-16T11:53:58.108 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68695.log 2024-09-16T11:53:58.111 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65150.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65150.log.gz 2024-09-16T11:53:58.111 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41485.log 2024-09-16T11:53:58.112 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68695.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68695.log.gz 2024-09-16T11:53:58.112 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57609.log 2024-09-16T11:53:58.112 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41485.log: 73.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41485.log.gz 2024-09-16T11:53:58.112 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41268.log 2024-09-16T11:53:58.115 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57609.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57609.log.gz 2024-09-16T11:53:58.115 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46658.log 2024-09-16T11:53:58.115 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41268.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41268.log.gz 2024-09-16T11:53:58.115 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65915.log 2024-09-16T11:53:58.116 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46658.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46658.log.gz 2024-09-16T11:53:58.116 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52219.log 2024-09-16T11:53:58.118 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65915.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65915.log.gz 2024-09-16T11:53:58.118 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.73796.log 2024-09-16T11:53:58.120 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52219.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52219.log.gz 2024-09-16T11:53:58.120 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57279.log 2024-09-16T11:53:58.122 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.73796.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.73796.log.gz 2024-09-16T11:53:58.122 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56359.log 2024-09-16T11:53:58.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57279.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57279.log.gz 2024-09-16T11:53:58.124 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49462.log 2024-09-16T11:53:58.125 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56359.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56359.log.gz 2024-09-16T11:53:58.125 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71448.log 2024-09-16T11:53:58.126 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49462.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49462.log.gz 2024-09-16T11:53:58.126 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37766.log 2024-09-16T11:53:58.128 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71448.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71448.log.gz 2024-09-16T11:53:58.128 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73593.log 2024-09-16T11:53:58.129 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.37766.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37766.log.gz 2024-09-16T11:53:58.129 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54839.log 2024-09-16T11:53:58.130 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73593.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73593.log.gz 2024-09-16T11:53:58.130 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64562.log 2024-09-16T11:53:58.132 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54839.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54839.log.gz 2024-09-16T11:53:58.133 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62169.log 2024-09-16T11:53:58.133 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64562.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64562.log.gz 2024-09-16T11:53:58.133 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68736.log 2024-09-16T11:53:58.134 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62169.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62169.log.gz 2024-09-16T11:53:58.134 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59542.log 2024-09-16T11:53:58.138 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68736.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68736.log.gz 2024-09-16T11:53:58.138 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51542.log 2024-09-16T11:53:58.139 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59542.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59542.log.gz 2024-09-16T11:53:58.139 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50698.log 2024-09-16T11:53:58.139 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51542.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51542.log.gz 2024-09-16T11:53:58.139 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27408.log 2024-09-16T11:53:58.142 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50698.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50698.log.gz 2024-09-16T11:53:58.142 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53995.log 2024-09-16T11:53:58.142 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.tmp-client.admin.27408.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27408.log.gz 2024-09-16T11:53:58.142 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53799.log 2024-09-16T11:53:58.143 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53995.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53995.log.gz 2024-09-16T11:53:58.143 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38759.log 2024-09-16T11:53:58.145 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53799.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53799.log.gz 2024-09-16T11:53:58.145 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67696.log 2024-09-16T11:53:58.145 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38759.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38759.log.gz 2024-09-16T11:53:58.147 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.67482.log 2024-09-16T11:53:58.147 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67696.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67696.log.gz 2024-09-16T11:53:58.147 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.46469.log 2024-09-16T11:53:58.154 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.67482.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.67482.log.gz 2024-09-16T11:53:58.154 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65956.log 2024-09-16T11:53:58.167 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.1.46469.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.47946.log.gz 2024-09-16T11:53:58.168 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-16T11:53:58.170 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65956.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65956.log.gz 2024-09-16T11:53:58.171 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45409.log 2024-09-16T11:53:58.187 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59816.log 2024-09-16T11:53:58.192 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45409.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45409.log.gz 2024-09-16T11:53:58.207 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46412.log 2024-09-16T11:53:58.217 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59816.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63147.log 2024-09-16T11:53:58.217 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59816.log.gz 2024-09-16T11:53:58.220 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52939.log 2024-09-16T11:53:58.221 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63147.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46412.log.gz 2024-09-16T11:53:58.227 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50619.log 2024-09-16T11:53:58.233 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52939.log: 87.4% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63147.log.gz 2024-09-16T11:53:58.236 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69110.log 2024-09-16T11:53:58.237 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50619.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50619.log.gz 2024-09-16T11:53:58.237 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52939.log.gz 2024-09-16T11:53:58.242 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73441.log.gz 2024-09-16T11:53:58.243 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45160.log 2024-09-16T11:53:58.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69110.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69110.log.gz 2024-09-16T11:53:58.244 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67482.log 2024-09-16T11:53:58.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45160.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.46469.log.gz 2024-09-16T11:53:58.246 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59903.log 2024-09-16T11:53:58.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67482.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67482.log.gz 2024-09-16T11:53:58.246 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37934.log 2024-09-16T11:53:58.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59903.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59903.log.gz 2024-09-16T11:53:58.261 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73255.log 2024-09-16T11:53:58.261 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37934.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37934.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74778.log 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73255.log: 88.6% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45160.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73255.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40746.log 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74778.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63882.log 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.74778.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68236.log 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40746.log: /var/log/ceph/cluster2-client.mirror.63882.log: 87.6% 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63882.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40746.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73486.log 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54777.log 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1%/var/log/ceph/cluster1-client.mirror.73486.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68236.log.gz 2024-09-16T11:53:58.262 INFO:teuthology.orchestra.run.smithi103.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70347.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.73486.log.gz 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51935.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70347.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54777.log.gz 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70347.log.gz 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69650.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64384.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69650.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69650.log.gz 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65355.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64384.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64384.log.gz 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48517.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65355.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65355.log.gz 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66824.log 2024-09-16T11:53:58.263 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48517.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48517.log.gz 2024-09-16T11:53:58.264 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55246.log 2024-09-16T11:53:58.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66824.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66824.log.gz 2024-09-16T11:53:58.315 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.49745.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65729.log 2024-09-16T11:53:58.315 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63978.log 2024-09-16T11:53:58.315 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65729.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55246.log.gz 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59720.log 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65729.log.gz 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.63978.log: --verbose -- /var/log/ceph/cluster1-client.mirror.59378.log 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63978.log.gz 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70325.log 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59378.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59720.log.gz 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32300.log 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70325.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70325.log.gz 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72934.log 2024-09-16T11:53:58.316 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4%/var/log/ceph/cluster2-client.admin.32300.log: -- replaced with /var/log/ceph/cluster1-client.mirror.59378.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32300.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69470.log 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66266.log 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72934.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46949.log 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66266.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69470.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46738.log 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46949.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71759.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46949.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58868.log 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66266.log.gz 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44747.log 2024-09-16T11:53:58.317 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47307.log 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46738.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.44747.log: /var/log/ceph/cluster2-client.mirror.58868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69357.log 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44747.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47307.log: gzip 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47307.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53947.log 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66163.log 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69357.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65222.log 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58868.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46309.log 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53947.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66163.log.gz 2024-09-16T11:53:58.318 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73463.log 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51876.log 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65222.log.gz 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41735.log 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51876.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73463.log.gz 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51876.log.gz 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51525.log 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38887.log 2024-09-16T11:53:58.319 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41735.log: /var/log/ceph/cluster2-client.mirror.51525.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47329.log.gz 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51525.log.gz 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41735.log.gz 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/cluster2-client.mirror.41755.log 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.38887.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38887.log.gz 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67836.log 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63219.log 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41755.log.gz 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.67836.log: gzip -5 --verbose 85.5% -- /var/log/ceph/cluster2-client.mirror.46246.log 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67836.log.gz 2024-09-16T11:53:58.320 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51755.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63219.log.gz 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64841.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58952.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51755.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46309.log.gz 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64841.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51755.log.gzgzip -5 --verbose -- 87.4% /var/log/ceph/cluster2-client.admin.74820.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64841.log.gz 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72038.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71717.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74820.log: 89.8% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74820.log.gz 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58952.log.gz 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48613.log 2024-09-16T11:53:58.321 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74483.log 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71717.log: /var/log/ceph/cluster2-client.mirror.48613.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71717.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48613.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62326.log 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40569.log 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72038.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74483.log: 0.0%/var/log/ceph/cluster2-client.mirror.62326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51381.log 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74483.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5%/var/log/ceph/cluster2-client.mirror.40569.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46246.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55945.log 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58597.log 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62326.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55945.log: /var/log/ceph/cluster2-client.mirror.51381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47781.log 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55945.log.gz 2024-09-16T11:53:58.322 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40569.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56142.log 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61031.log 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58597.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51381.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47781.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64913.log 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56142.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67445.log 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.64913.log: --verbose -- 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61031.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr: /var/log/ceph/cluster2-client.mirror.67052.log 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67445.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64913.log.gz 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61996.log 2024-09-16T11:53:58.323 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62862.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61996.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67052.log.gz 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43442.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61996.log.gz 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.72326.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62862.log: /var/log/ceph/cluster2-client.mirror.43442.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67445.log.gz 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43442.log.gz 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49138.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62862.log.gz 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.2.72326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75156.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62882.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:gzip 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49138.log.gz -5 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.69942.log 2024-09-16T11:53:58.324 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75156.log: gzip/var/log/ceph/cluster2-client.mirror.62882.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66762.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75156.log.gz 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69942.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62882.log.gz 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69942.log.gz 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56705.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41959.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66762.log: /var/log/ceph/cluster2-client.mirror.56705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58502.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41959.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56705.log.gz 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63609.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58502.log: 90.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58502.log.gz 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66762.log.gz 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72646.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60081.log 2024-09-16T11:53:58.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72646.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63609.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.72326.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71112.log 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64473.log 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71112.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60081.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71112.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60588.log 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69717.log 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 91.5%/var/log/ceph/cluster1-client.mirror.60588.log: -- replaced with /var/log/ceph/cluster2-client.mirror.72646.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41959.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64473.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60588.log.gz 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66907.log 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48974.log 2024-09-16T11:53:58.326 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69717.log.gz 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48151.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48974.log: gzip 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66907.log.gz 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46932.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54417.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46932.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48151.log.gz 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46932.log.gz 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.47132.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53056.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54417.log.gz 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72204.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47132.log.gz 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63725.log 2024-09-16T11:53:58.327 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41031.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72204.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71638.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41031.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53056.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63725.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr: 73.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41031.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73341.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37808.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.73341.log: -- replaced with /var/log/ceph/cluster2-client.mirror.71638.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58799.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73341.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51150.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.37808.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37808.log.gz 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40543.log 2024-09-16T11:53:58.328 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58799.log.gz 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50278.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51150.log: /var/log/ceph/cluster1-client.admin.40543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55540.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50278.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.40543.log.gz 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50278.log.gz 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51150.log.gz 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46031.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52535.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66308.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52535.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52535.log.gz 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48755.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55540.log.gz 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57629.log 2024-09-16T11:53:58.329 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48755.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48755.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66308.log.gz 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55873.log 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71679.log 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57629.log.gz 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43019.log 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55873.log.gz 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59792.log 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43019.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46031.log.gz 2024-09-16T11:53:58.330 INFO:teuthology.orchestra.run.smithi103.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71679.log.gz 2024-09-16T11:53:58.351 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67947.log 2024-09-16T11:53:58.351 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53481.log 2024-09-16T11:53:58.351 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67947.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59792.log.gz 2024-09-16T11:53:58.351 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67947.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55198.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53481.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53481.log.gz 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72791.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50935.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55198.log.gz 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5%/var/log/ceph/cluster2-client.mirror.72791.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43019.log.gz 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68339.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50935.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72791.log.gz 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50735.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50935.log.gz/var/log/ceph/cluster2-client.mirror.68339.log: 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53309.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56625.log 2024-09-16T11:53:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50735.log: /var/log/ceph/cluster2-client.mirror.53309.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50735.log.gz 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68339.log.gz 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- 92.7% /var/log/ceph/cluster2-client.mirror.47366.log 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53309.log.gz 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47386.log 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69852.log 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47366.log: /var/log/ceph/cluster2-client.mirror.47386.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47386.log.gz 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47366.log.gz 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56625.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48712.log 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60705.log 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69852.log.gz 2024-09-16T11:53:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72248.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48712.log.gz 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65451.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60705.log: /var/log/ceph/cluster2-client.mirror.72248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66680.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65451.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72248.log.gz 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65451.log.gz 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67926.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44956.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67926.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66680.log.gz 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60705.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67926.log.gz 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51965.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-16T11:53:58.354 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44685.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51965.log.gz 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-mon.a.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44956.log.gz 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68997.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58171.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68997.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44685.log.gz 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68997.log.gz 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71159.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.51621.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58171.log.gz 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71159.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71159.log.gz 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57125.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.3.51621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59632.log 2024-09-16T11:53:58.355 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57125.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57125.log.gz 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42488.log 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59632.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59632.log.gz 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54176.log 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42488.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42488.log.gz 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67300.log 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54176.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42664.log.gz 2024-09-16T11:53:58.356 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54176.log.gz 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46991.log 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45945.log 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67300.log.gz 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46558.log 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45945.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45945.log.gz 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48734.log 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46558.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46558.log.gz 2024-09-16T11:53:58.361 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49346.log 2024-09-16T11:53:58.372 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45452.log 2024-09-16T11:53:58.373 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49346.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49346.log.gz 2024-09-16T11:53:58.389 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.51621.log.gz 2024-09-16T11:53:58.390 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60499.log 2024-09-16T11:53:58.396 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45452.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45452.log.gz 2024-09-16T11:53:58.396 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64821.log 2024-09-16T11:53:58.397 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60499.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60499.log.gz 2024-09-16T11:53:58.397 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48692.log 2024-09-16T11:53:58.399 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64821.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64821.log.gz 2024-09-16T11:53:58.399 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73971.log 2024-09-16T11:53:58.400 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48692.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48692.log.gz 2024-09-16T11:53:58.400 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72747.log 2024-09-16T11:53:58.404 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73971.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73971.log.gz 2024-09-16T11:53:58.404 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.71797.log 2024-09-16T11:53:58.405 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72747.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72747.log.gz 2024-09-16T11:53:58.405 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63930.log 2024-09-16T11:53:58.412 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.1.71797.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.71797.log.gz 2024-09-16T11:53:58.412 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39105.log 2024-09-16T11:53:58.413 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63930.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63930.log.gz 2024-09-16T11:53:58.413 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65379.log 2024-09-16T11:53:58.415 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39105.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39105.log.gz 2024-09-16T11:53:58.415 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53329.log 2024-09-16T11:53:58.416 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.65379.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65379.log.gz 2024-09-16T11:53:58.416 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54998.log 2024-09-16T11:53:58.430 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53329.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53329.log.gz 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66556.log 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54998.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54998.log.gz 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55704.log 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66556.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66556.log.gz 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39032.log 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55704.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55704.log.gz 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70077.log 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39032.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39032.log.gz 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67988.log 2024-09-16T11:53:58.431 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70077.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70077.log.gz 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32522.log 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67988.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67988.log.gz 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57412.log 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.32522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32522.log.gz 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59881.log 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57412.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57412.log.gz 2024-09-16T11:53:58.432 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45024.log 2024-09-16T11:53:58.436 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59881.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59881.log.gz 2024-09-16T11:53:58.436 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48792.log 2024-09-16T11:53:58.436 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45024.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45024.log.gz 2024-09-16T11:53:58.436 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51401.log 2024-09-16T11:53:58.436 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48792.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48792.log.gz 2024-09-16T11:53:58.437 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54293.log 2024-09-16T11:53:58.437 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51401.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51401.log.gz 2024-09-16T11:53:58.437 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67010.log 2024-09-16T11:53:58.438 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54293.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54293.log.gz 2024-09-16T11:53:58.438 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.44158.log 2024-09-16T11:53:58.441 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67010.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67010.log.gz 2024-09-16T11:53:58.442 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42887.log 2024-09-16T11:53:58.452 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.2.44158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57038.log 2024-09-16T11:53:58.453 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42887.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42887.log.gz 2024-09-16T11:53:58.466 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50678.log 2024-09-16T11:53:58.467 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57038.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57038.log.gz 2024-09-16T11:53:58.467 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42416.log 2024-09-16T11:53:58.473 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54618.log 2024-09-16T11:53:58.480 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70415.log 2024-09-16T11:53:58.487 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54618.log: 87.4%/var/log/ceph/cluster2-client.mirror.42416.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42416.log.gz 2024-09-16T11:53:58.488 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40362.log 2024-09-16T11:53:58.495 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70415.log: 86.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54618.log.gz 2024-09-16T11:53:58.495 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56645.log 2024-09-16T11:53:58.498 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40362.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40362.log.gz 2024-09-16T11:53:58.502 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70415.log.gz 2024-09-16T11:53:58.502 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44103.log 2024-09-16T11:53:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60057.log 2024-09-16T11:53:58.516 INFO:teuthology.orchestra.run.smithi103.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56645.log.gz 2024-09-16T11:53:58.522 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44103.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51935.log.gz 2024-09-16T11:53:58.523 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68674.log 2024-09-16T11:53:58.523 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60057.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60057.log.gz 2024-09-16T11:53:58.537 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75240.log 2024-09-16T11:53:58.538 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68674.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68674.log.gz 2024-09-16T11:53:58.551 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67217.log 2024-09-16T11:53:58.551 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75240.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75240.log.gz 2024-09-16T11:53:58.562 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48734.log.gz 2024-09-16T11:53:58.562 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66411.log 2024-09-16T11:53:58.562 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67217.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67217.log.gz 2024-09-16T11:53:58.589 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55222.log 2024-09-16T11:53:58.589 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66411.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66411.log.gz 2024-09-16T11:53:58.602 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53778.log 2024-09-16T11:53:58.602 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55222.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55222.log.gz 2024-09-16T11:53:58.605 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.44158.log.gz 2024-09-16T11:53:58.607 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73613.log 2024-09-16T11:53:58.607 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53778.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53778.log.gz 2024-09-16T11:53:58.607 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71341.log 2024-09-16T11:53:58.611 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73613.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73613.log.gz 2024-09-16T11:53:58.611 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59113.log 2024-09-16T11:53:58.611 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71341.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71341.log.gz 2024-09-16T11:53:58.611 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38306.log 2024-09-16T11:53:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59113.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59113.log.gz 2024-09-16T11:53:58.613 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63306.log 2024-09-16T11:53:58.615 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38306.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38306.log.gz 2024-09-16T11:53:58.615 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43784.log 2024-09-16T11:53:58.616 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63306.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63306.log.gz 2024-09-16T11:53:58.616 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57320.log 2024-09-16T11:53:58.619 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43784.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43784.log.gz 2024-09-16T11:53:58.619 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60412.log 2024-09-16T11:53:58.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57320.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57320.log.gz 2024-09-16T11:53:58.623 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45601.log 2024-09-16T11:53:58.623 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60412.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60412.log.gz 2024-09-16T11:53:58.623 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46137.log 2024-09-16T11:53:58.628 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45601.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45601.log.gz 2024-09-16T11:53:58.629 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71888.log 2024-09-16T11:53:58.635 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.46137.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46137.log.gz 2024-09-16T11:53:58.635 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59768.log 2024-09-16T11:53:58.636 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71888.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71888.log.gz 2024-09-16T11:53:58.636 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38998.log 2024-09-16T11:53:58.639 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59768.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59768.log.gz 2024-09-16T11:53:58.639 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48218.log 2024-09-16T11:53:58.639 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38998.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38998.log.gz 2024-09-16T11:53:58.639 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61184.log 2024-09-16T11:53:58.640 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48218.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48218.log.gz 2024-09-16T11:53:58.640 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70595.log 2024-09-16T11:53:58.643 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61184.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61184.log.gz 2024-09-16T11:53:58.643 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-16T11:53:58.643 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70595.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70595.log.gz 2024-09-16T11:53:58.643 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68590.log 2024-09-16T11:53:58.654 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57183.log 2024-09-16T11:53:58.661 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68590.log: 90.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69335.log 2024-09-16T11:53:58.661 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68590.log.gz 2024-09-16T11:53:58.674 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68133.log 2024-09-16T11:53:58.674 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57183.log.gz 2024-09-16T11:53:58.674 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69335.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69335.log.gz 2024-09-16T11:53:58.688 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61800.log 2024-09-16T11:53:58.689 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68133.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68133.log.gz 2024-09-16T11:53:58.690 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50678.log.gz 2024-09-16T11:53:58.690 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43996.log 2024-09-16T11:53:58.692 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61800.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61800.log.gz 2024-09-16T11:53:58.692 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73361.log 2024-09-16T11:53:58.694 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43996.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48974.log.gz 2024-09-16T11:53:58.695 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57529.log 2024-09-16T11:53:58.698 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73361.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73361.log.gz 2024-09-16T11:53:58.698 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46392.log 2024-09-16T11:53:58.698 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43996.log.gz 2024-09-16T11:53:58.699 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57529.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57529.log.gz 2024-09-16T11:53:58.700 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49304.log 2024-09-16T11:53:58.700 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.52437.log 2024-09-16T11:53:58.701 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46392.log: /var/log/ceph/cluster2-client.mirror.49304.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46392.log.gz 2024-09-16T11:53:58.701 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49304.log.gz 2024-09-16T11:53:58.701 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57207.log 2024-09-16T11:53:58.703 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.0.52437.log: /var/log/ceph/cluster1-client.mirror.53240.log 2024-09-16T11:53:58.703 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57207.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57207.log.gz 2024-09-16T11:53:58.703 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67279.log 2024-09-16T11:53:58.705 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53240.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53240.log.gz 2024-09-16T11:53:58.705 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45180.log 2024-09-16T11:53:58.706 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67279.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67279.log.gz 2024-09-16T11:53:58.706 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55403.log 2024-09-16T11:53:58.712 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45180.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45180.log.gz 2024-09-16T11:53:58.712 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62930.log 2024-09-16T11:53:58.713 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55403.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55403.log.gz 2024-09-16T11:53:58.714 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55562.log 2024-09-16T11:53:58.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62930.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62930.log.gz 2024-09-16T11:53:58.715 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61880.log 2024-09-16T11:53:58.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55562.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55562.log.gz 2024-09-16T11:53:58.716 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51982.log 2024-09-16T11:53:58.717 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61880.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61880.log.gz 2024-09-16T11:53:58.718 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42911.log 2024-09-16T11:53:58.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51982.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51982.log.gz 2024-09-16T11:53:58.720 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42320.log 2024-09-16T11:53:58.720 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42911.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42911.log.gz 2024-09-16T11:53:58.720 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69672.log 2024-09-16T11:53:58.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.42320.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42320.log.gz 2024-09-16T11:53:58.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74694.log 2024-09-16T11:53:58.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69672.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69672.log.gz 2024-09-16T11:53:58.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58534.log 2024-09-16T11:53:58.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74694.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74694.log.gz 2024-09-16T11:53:58.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48834.log 2024-09-16T11:53:58.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58534.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58534.log.gz 2024-09-16T11:53:58.729 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74862.log 2024-09-16T11:53:58.729 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48834.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48834.log.gz 2024-09-16T11:53:58.729 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57701.log 2024-09-16T11:53:58.729 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74862.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74862.log.gz 2024-09-16T11:53:58.729 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53971.log 2024-09-16T11:53:58.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57701.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57701.log.gz 2024-09-16T11:53:58.730 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57746.log 2024-09-16T11:53:58.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53971.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53971.log.gz 2024-09-16T11:53:58.730 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66494.log 2024-09-16T11:53:58.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57746.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57746.log.gz 2024-09-16T11:53:58.732 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61438.log 2024-09-16T11:53:58.732 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66494.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66494.log.gz 2024-09-16T11:53:58.733 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71426.log 2024-09-16T11:53:58.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61438.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61438.log.gz 2024-09-16T11:53:58.734 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40093.log 2024-09-16T11:53:58.736 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71426.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71426.log.gz 2024-09-16T11:53:58.736 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47200.log 2024-09-16T11:53:58.738 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40093.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40093.log.gz 2024-09-16T11:53:58.738 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71176.log 2024-09-16T11:53:58.740 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47200.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47200.log.gz 2024-09-16T11:53:58.740 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64159.log 2024-09-16T11:53:58.741 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71176.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71176.log.gz 2024-09-16T11:53:58.741 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73528.log 2024-09-16T11:53:58.741 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64159.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64159.log.gz 2024-09-16T11:53:58.744 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.50912.log 2024-09-16T11:53:58.744 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73528.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73528.log.gz 2024-09-16T11:53:58.744 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63376.log 2024-09-16T11:53:58.744 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.0.50912.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46991.log.gz 2024-09-16T11:53:58.744 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58727.log 2024-09-16T11:53:58.745 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63376.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63376.log.gz 2024-09-16T11:53:58.745 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41340.log 2024-09-16T11:53:58.751 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58727.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58727.log.gz 2024-09-16T11:53:58.751 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 93.4% --verbose -- /var/log/ceph/cluster1-client.mirror.69132.log 2024-09-16T11:53:58.751 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.52437.log.gz 2024-09-16T11:53:58.751 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51213.log 2024-09-16T11:53:58.751 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69132.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69132.log.gz 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41340.log.gz 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73671.log 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70100.log 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51213.log.gz 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64183.log 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70100.log: 86.7% 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73671.log.gz 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70100.log.gz 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40431.log 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52853.log 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64183.log.gz 2024-09-16T11:53:58.752 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73909.log 2024-09-16T11:53:58.753 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52853.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52853.log.gz 2024-09-16T11:53:58.753 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59337.log 2024-09-16T11:53:58.757 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73909.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73909.log.gz 2024-09-16T11:53:58.757 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50533.log 2024-09-16T11:53:58.757 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59337.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59337.log.gz 2024-09-16T11:53:58.757 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72897.log 2024-09-16T11:53:58.758 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50533.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50533.log.gz 2024-09-16T11:53:58.758 INFO:teuthology.orchestra.run.smithi103.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40431.log.gz 2024-09-16T11:53:58.758 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74552.log 2024-09-16T11:53:58.758 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68443.log 2024-09-16T11:53:58.758 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.74552.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74552.log.gz 2024-09-16T11:53:58.758 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67675.log 2024-09-16T11:53:58.760 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68443.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68443.log.gz 2024-09-16T11:53:58.760 INFO:teuthology.orchestra.run.smithi103.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72897.log.gz 2024-09-16T11:53:58.760 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71320.log 2024-09-16T11:53:58.760 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32340.log 2024-09-16T11:53:58.760 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71320.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71320.log.gz 2024-09-16T11:53:58.762 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64889.log 2024-09-16T11:53:58.762 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.32340.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32340.log.gz 2024-09-16T11:53:58.762 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53076.log 2024-09-16T11:53:58.762 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67675.log.gz 2024-09-16T11:53:58.762 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68652.log 2024-09-16T11:53:58.762 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64889.log.gz/var/log/ceph/cluster1-client.mirror.53076.log: 2024-09-16T11:53:58.764 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63517.log 2024-09-16T11:53:58.764 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68652.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53076.log.gz 2024-09-16T11:53:58.764 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47183.log 2024-09-16T11:53:58.767 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63517.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68652.log.gz 2024-09-16T11:53:58.767 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44363.log 2024-09-16T11:53:58.767 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47183.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47183.log.gz 2024-09-16T11:53:58.767 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45836.log 2024-09-16T11:53:58.768 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63517.log.gz/var/log/ceph/cluster1-client.admin.44363.log: 2024-09-16T11:53:58.768 INFO:teuthology.orchestra.run.smithi103.stderr:gzip 87.9% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49723.log 2024-09-16T11:53:58.768 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44363.log.gz 2024-09-16T11:53:58.768 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67383.log 2024-09-16T11:53:58.769 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49723.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49723.log.gz 2024-09-16T11:53:58.769 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.64710.log 2024-09-16T11:53:58.771 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67383.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67383.log.gz 2024-09-16T11:53:58.771 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46638.log 2024-09-16T11:53:58.773 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.0.64710.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45836.log.gz 2024-09-16T11:53:58.774 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69582.log 2024-09-16T11:53:58.774 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46638.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46638.log.gz 2024-09-16T11:53:58.775 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40972.log 2024-09-16T11:53:58.776 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69582.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69582.log.gz 2024-09-16T11:53:58.776 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43670.log 2024-09-16T11:53:58.777 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40972.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40972.log.gz 2024-09-16T11:53:58.778 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73041.log 2024-09-16T11:53:58.779 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43670.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43670.log.gz 2024-09-16T11:53:58.779 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62121.log 2024-09-16T11:53:58.780 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73041.log: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73041.log.gz 2024-09-16T11:53:58.780 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69177.log 2024-09-16T11:53:58.782 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62121.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62121.log.gz 2024-09-16T11:53:58.782 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64115.log 2024-09-16T11:53:58.783 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69177.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69177.log.gz 2024-09-16T11:53:58.783 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72227.log 2024-09-16T11:53:58.784 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64115.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64115.log.gz 2024-09-16T11:53:58.785 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47287.log 2024-09-16T11:53:58.785 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72227.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72227.log.gz 2024-09-16T11:53:58.786 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69605.log 2024-09-16T11:53:58.787 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47287.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47287.log.gz 2024-09-16T11:53:58.788 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54552.log 2024-09-16T11:53:58.789 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69605.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69605.log.gz 2024-09-16T11:53:58.789 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49405.log 2024-09-16T11:53:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54552.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54552.log.gz 2024-09-16T11:53:58.791 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73738.log 2024-09-16T11:53:58.799 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49405.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.50912.log.gz 2024-09-16T11:53:58.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42976.log 2024-09-16T11:53:58.801 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73738.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73738.log.gz 2024-09-16T11:53:58.807 INFO:teuthology.orchestra.run.smithi103.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.64710.log.gz 2024-09-16T11:53:58.808 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52556.log 2024-09-16T11:53:58.808 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.42976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.43303.log 2024-09-16T11:53:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52556.log.gz 2024-09-16T11:53:58.813 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42976.log.gz 2024-09-16T11:53:58.814 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43240.log 2024-09-16T11:53:58.814 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.1.43303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73211.log 2024-09-16T11:53:58.815 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.43240.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43240.log.gz 2024-09-16T11:53:58.829 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39794.log 2024-09-16T11:53:58.829 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73211.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73211.log.gz 2024-09-16T11:53:58.831 INFO:teuthology.orchestra.run.smithi103.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.43303.log.gz 2024-09-16T11:53:58.832 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41875.log 2024-09-16T11:53:58.832 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60612.log 2024-09-16T11:53:58.833 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41875.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41875.log.gz 2024-09-16T11:53:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72289.log 2024-09-16T11:53:58.846 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60612.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60612.log.gz 2024-09-16T11:53:58.858 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65791.log 2024-09-16T11:53:58.861 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72289.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72289.log.gz 2024-09-16T11:53:58.871 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47220.log 2024-09-16T11:53:59.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65791.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65791.log.gz 2024-09-16T11:53:59.265 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62280.log 2024-09-16T11:53:59.265 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47220.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47220.log.gz 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49197.log 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62280.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62280.log.gz 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40866.log 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49197.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49197.log.gz 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61298.log 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.40866.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40866.log.gz 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69965.log 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61298.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61298.log.gz 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54313.log 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69965.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69965.log.gz 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48396.log 2024-09-16T11:53:59.266 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54313.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54313.log.gz 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59265.log 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48396.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48396.log.gz 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41316.log 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59265.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59265.log.gz 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52915.log 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41316.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41316.log.gz 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66328.log 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52915.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52915.log.gz 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52160.log 2024-09-16T11:53:59.267 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66328.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66328.log.gz 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68319.log 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52160.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52160.log.gz 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38306.log 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68319.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68319.log.gz 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41448.log 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.0.38306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59197.log 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59961.log 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54393.log 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59961.log: 87.4% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59197.log.gz 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr: 92.3% 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41448.log.gz 2024-09-16T11:53:59.268 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.38306.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59961.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73651.log 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54393.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54393.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43731.log 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73651.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73651.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62037.log 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.43731.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.43731.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59241.log 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62037.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62037.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48438.log 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59241.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59241.log.gz 2024-09-16T11:53:59.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66969.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48438.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48438.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54272.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66969.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66969.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74202.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54272.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54272.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40516.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74202.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74202.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42642.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40516.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40516.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63812.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42642.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42642.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53841.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63812.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63812.log.gz 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64231.log 2024-09-16T11:53:59.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53841.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53841.log.gz 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65770.log 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64231.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64231.log.gz 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53730.log 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65770.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65770.log.gz 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50715.log 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53730.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53730.log.gz 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42293.log 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50715.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50715.log.gz 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43583.log 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.42293.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42293.log.gz 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69492.log 2024-09-16T11:53:59.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43583.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43583.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75198.log 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69492.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69492.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61755.log 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.75198.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75198.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61506.log 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61755.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61755.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63123.log 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61506.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61506.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56604.log 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.63123.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63123.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61007.log 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56604.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56604.log.gz 2024-09-16T11:53:59.272 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71594.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61007.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61007.log.gz 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55499.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71594.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71594.log.gz 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68215.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55499.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55499.log.gz 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49703.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68215.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68215.log.gz 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71276.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49703.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49703.log.gz 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44157.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71276.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71276.log.gz 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54335.log 2024-09-16T11:53:59.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.44157.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44157.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72540.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54335.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54335.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74268.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72540.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72540.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70640.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.74268.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74268.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48475.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70640.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70640.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47665.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48475.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48475.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51463.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47665.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47665.log.gz 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74135.log 2024-09-16T11:53:59.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51463.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51463.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67717.log 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74135.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74135.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52670.log 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67717.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67717.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49160.log 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52670.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52670.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46758.log 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49160.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49405.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48932.log 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46758.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46758.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71406.log 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48932.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48932.log.gz 2024-09-16T11:53:59.275 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.44157.log 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71406.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71406.log.gz 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.38399.log 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.3.44157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58099.log 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.2.38399.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.38399.log.gz 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63424.log 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59840.log 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58099.log.gz 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63424.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63424.log.gz 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65853.log 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59840.log: 87.7% 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.44157.log.gz 2024-09-16T11:53:59.276 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59840.log.gz 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57460.log 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65853.log: 88.0% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39794.log.gz 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65853.log.gz 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65646.log 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58575.log 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57460.log.gz 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41672.log 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58575.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58575.log.gz 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65646.log.gz 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53570.log 2024-09-16T11:53:59.277 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59313.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53570.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41672.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50046.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59313.log: 87.4% 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53570.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59313.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64065.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48851.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50046.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57571.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48851.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64065.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41981.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48851.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61900.log 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41981.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57571.log.gz 2024-09-16T11:53:59.278 INFO:teuthology.orchestra.run.smithi103.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41981.log.gz 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.44837.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47946.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61900.log.gz 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.2.44837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51192.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47946.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44103.log.gz 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72770.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.55035.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72770.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72770.log.gz 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45252.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.3.55035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39125.log 2024-09-16T11:53:59.279 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45252.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45252.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70617.log 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39125.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39125.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.44837.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47946.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55270.log 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70617.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70617.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68050.log 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55270.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55270.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40290.log 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68050.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68050.log.gz 2024-09-16T11:53:59.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55769.log 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40290.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40290.log.gz 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65032.log 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55769.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55769.log.gz 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72076.log 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65032.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65032.log.gz 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69447.log 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72076.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72076.log.gz 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60033.log 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69447.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69447.log.gz 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55897.log 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60033.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60033.log.gz 2024-09-16T11:53:59.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64023.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55897.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55897.log.gz 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72812.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64023.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64023.log.gz 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51913.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.72812.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72812.log.gz 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50777.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51913.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51913.log.gz 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62370.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50777.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50777.log.gz 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47487.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62370.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62370.log.gz 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41385.log 2024-09-16T11:53:59.282 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47487.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.55035.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58478.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.41385.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41385.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45687.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58478.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58478.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57484.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45687.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45687.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59518.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57484.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57484.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43490.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59518.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59518.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41405.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43490.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43490.log.gz 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56922.log 2024-09-16T11:53:59.283 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41405.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41405.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38691.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56922.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56922.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59068.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.38691.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.38691.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73930.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59068.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59068.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63448.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73930.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73930.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65082.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63448.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63448.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40137.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65082.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65082.log.gz 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59289.log 2024-09-16T11:53:59.284 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.40137.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40137.log.gz 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55451.log 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59289.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59289.log.gz 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58406.log 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55451.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55451.log.gz 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73759.log 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.58406.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58406.log.gz 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63906.log 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.73759.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73759.log.gz 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51562.log 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63906.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63906.log.gz 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47724.log 2024-09-16T11:53:59.285 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51562.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51562.log.gz 2024-09-16T11:53:59.286 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66142.log 2024-09-16T11:53:59.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54935.log 2024-09-16T11:53:59.286 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66142.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66142.log.gz 2024-09-16T11:53:59.287 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37472.log 2024-09-16T11:53:59.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37010.log 2024-09-16T11:53:59.288 INFO:teuthology.orchestra.run.smithi103.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54935.log.gz 2024-09-16T11:53:59.288 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37472.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37472.log.gz 2024-09-16T11:53:59.294 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44600.log 2024-09-16T11:53:59.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37010.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37010.log.gz 2024-09-16T11:53:59.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66287.log 2024-09-16T11:53:59.304 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44600.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51192.log.gz 2024-09-16T11:53:59.304 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67425.log 2024-09-16T11:53:59.305 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66287.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66287.log.gz 2024-09-16T11:53:59.309 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44600.log.gz 2024-09-16T11:53:59.309 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47408.log 2024-09-16T11:53:59.311 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67425.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67425.log.gz 2024-09-16T11:53:59.311 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72059.log 2024-09-16T11:53:59.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72856.log 2024-09-16T11:53:59.325 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72059.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72059.log.gz 2024-09-16T11:53:59.339 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41364.log 2024-09-16T11:53:59.339 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72856.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72856.log.gz 2024-09-16T11:53:59.347 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69875.log 2024-09-16T11:53:59.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.60181.log 2024-09-16T11:53:59.358 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69875.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69875.log.gz 2024-09-16T11:53:59.360 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41364.log.gz 2024-09-16T11:53:59.368 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64002.log 2024-09-16T11:53:59.377 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.2.60181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74157.log 2024-09-16T11:53:59.386 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64002.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56267.log 2024-09-16T11:53:59.390 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74157.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64002.log.gz 2024-09-16T11:53:59.395 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50814.log 2024-09-16T11:53:59.404 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56267.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53264.log 2024-09-16T11:53:59.408 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50814.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56267.log.gz 2024-09-16T11:53:59.417 INFO:teuthology.orchestra.run.smithi103.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50814.log.gz 2024-09-16T11:53:59.421 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67593.log 2024-09-16T11:53:59.422 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53264.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53264.log.gz 2024-09-16T11:53:59.424 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.60181.log.gz 2024-09-16T11:53:59.433 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51856.log 2024-09-16T11:53:59.434 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67593.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67593.log.gz 2024-09-16T11:53:59.454 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71485.log 2024-09-16T11:53:59.465 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70370.log 2024-09-16T11:53:59.465 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71485.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71485.log.gz 2024-09-16T11:53:59.473 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55383.log 2024-09-16T11:53:59.486 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70370.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70370.log.gz 2024-09-16T11:53:59.493 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61595.log 2024-09-16T11:53:59.494 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55383.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55383.log.gz 2024-09-16T11:53:59.511 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71997.log 2024-09-16T11:53:59.514 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61595.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61595.log.gz 2024-09-16T11:53:59.520 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53754.log 2024-09-16T11:53:59.523 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.71997.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49160.log.gz 2024-09-16T11:53:59.526 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61617.log 2024-09-16T11:53:59.526 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53754.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53754.log.gz 2024-09-16T11:53:59.527 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66039.log 2024-09-16T11:53:59.527 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61617.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61617.log.gz 2024-09-16T11:53:59.527 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32439.log 2024-09-16T11:53:59.530 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66039.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66039.log.gz 2024-09-16T11:53:59.530 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37850.log 2024-09-16T11:53:59.530 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.32439.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32439.log.gz 2024-09-16T11:53:59.530 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61094.log 2024-09-16T11:53:59.530 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.37850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37850.log.gz 2024-09-16T11:53:59.530 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53389.log 2024-09-16T11:53:59.531 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61094.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61094.log.gz 2024-09-16T11:53:59.532 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68777.log 2024-09-16T11:53:59.534 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.53389.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53389.log.gz 2024-09-16T11:53:59.534 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65335.log 2024-09-16T11:53:59.534 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68777.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71997.log.gz 2024-09-16T11:53:59.535 INFO:teuthology.orchestra.run.smithi103.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68777.log.gz 2024-09-16T11:53:59.536 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56749.log 2024-09-16T11:53:59.537 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65335.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65335.log.gz 2024-09-16T11:53:59.538 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47840.log 2024-09-16T11:53:59.538 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56749.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56749.log.gz 2024-09-16T11:53:59.539 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66246.log 2024-09-16T11:53:59.539 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.47840.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47840.log.gz 2024-09-16T11:53:59.540 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49012.log 2024-09-16T11:53:59.540 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66246.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66246.log.gz 2024-09-16T11:53:59.541 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70145.log 2024-09-16T11:53:59.553 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.49012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68071.log 2024-09-16T11:53:59.554 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70145.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70145.log.gz 2024-09-16T11:53:59.558 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49012.log.gz 2024-09-16T11:53:59.568 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53505.log 2024-09-16T11:53:59.568 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68071.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68071.log.gz 2024-09-16T11:53:59.578 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59400.log 2024-09-16T11:53:59.590 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53505.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53505.log.gz 2024-09-16T11:53:59.591 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51584.log 2024-09-16T11:53:59.593 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.59400.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59400.log.gz 2024-09-16T11:53:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37346.log 2024-09-16T11:53:59.622 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46598.log 2024-09-16T11:53:59.622 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37346.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37346.log.gz 2024-09-16T11:53:59.630 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.39982.log 2024-09-16T11:53:59.638 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46598.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46598.log.gz 2024-09-16T11:53:59.649 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74504.log 2024-09-16T11:53:59.651 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.2.39982.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.39982.log.gz 2024-09-16T11:53:59.659 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48339.log 2024-09-16T11:53:59.660 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.74504.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74504.log.gz 2024-09-16T11:53:59.667 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47724.log.gz 2024-09-16T11:53:59.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.mon.a has been restored 2024-09-16T11:53:59.675 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61731.log 2024-09-16T11:53:59.696 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45816.log 2024-09-16T11:53:59.697 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61731.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61731.log.gz 2024-09-16T11:53:59.703 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47487.log.gz 2024-09-16T11:53:59.704 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62394.log 2024-09-16T11:53:59.710 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.45816.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45816.log.gz 2024-09-16T11:53:59.710 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64710.log 2024-09-16T11:53:59.711 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62394.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62394.log.gz 2024-09-16T11:53:59.711 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36917.log 2024-09-16T11:53:59.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.64710.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64710.log.gz 2024-09-16T11:53:59.719 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47011.log 2024-09-16T11:53:59.719 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.36917.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.36917.log.gz 2024-09-16T11:53:59.722 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37133.log 2024-09-16T11:53:59.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47011.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47011.log.gz 2024-09-16T11:53:59.722 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52044.log 2024-09-16T11:53:59.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.37133.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37133.log.gz 2024-09-16T11:53:59.722 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69627.log 2024-09-16T11:53:59.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52044.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52044.log.gz 2024-09-16T11:53:59.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52182.log 2024-09-16T11:53:59.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69627.log: 93.8% 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69627.log.gz 2024-09-16T11:53:59.724 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47408.log.gz 2024-09-16T11:53:59.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56404.log 2024-09-16T11:53:59.733 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73886.log 2024-09-16T11:53:59.747 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56404.log: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48772.log 2024-09-16T11:53:59.747 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56404.log.gz 2024-09-16T11:53:59.747 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.73886.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73886.log.gz 2024-09-16T11:53:59.761 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39015.log 2024-09-16T11:53:59.763 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48772.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48772.log.gz 2024-09-16T11:53:59.763 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-16T11:53:59.763 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.39015.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39015.log.gz 2024-09-16T11:53:59.777 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56488.log 2024-09-16T11:53:59.782 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2.log: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-16T11:53:59.782 INFO:teuthology.orchestra.run.smithi103.stderr: 86.5% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-16T11:53:59.783 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43213.log 2024-09-16T11:53:59.785 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56488.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56488.log.gz 2024-09-16T11:53:59.785 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48576.log 2024-09-16T11:53:59.786 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.43213.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43213.log.gz 2024-09-16T11:53:59.786 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54152.log 2024-09-16T11:53:59.798 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51483.log 2024-09-16T11:53:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.54152.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54152.log.gz 2024-09-16T11:53:59.812 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74289.log 2024-09-16T11:53:59.813 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51483.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51483.log.gz 2024-09-16T11:53:59.813 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63195.log 2024-09-16T11:53:59.814 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.74289.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74289.log.gz 2024-09-16T11:53:59.827 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68257.log 2024-09-16T11:53:59.828 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63195.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63195.log.gz 2024-09-16T11:53:59.842 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56311.log 2024-09-16T11:53:59.842 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68257.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68257.log.gz 2024-09-16T11:53:59.843 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46469.log 2024-09-16T11:53:59.849 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56311.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45000.log 2024-09-16T11:53:59.849 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56311.log.gz 2024-09-16T11:53:59.855 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62217.log 2024-09-16T11:53:59.866 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45000.log: /var/log/ceph/cluster1-client.mirror.46469.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46469.log.gz 2024-09-16T11:53:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51777.log 2024-09-16T11:53:59.869 INFO:teuthology.orchestra.run.smithi103.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45000.log.gz 2024-09-16T11:53:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62217.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62217.log.gz 2024-09-16T11:53:59.876 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59494.log 2024-09-16T11:53:59.885 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.51777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70775.log 2024-09-16T11:53:59.893 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.59494.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59494.log.gz 2024-09-16T11:53:59.898 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41148.log 2024-09-16T11:53:59.898 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70775.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70775.log.gz 2024-09-16T11:53:59.905 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52298.log 2024-09-16T11:53:59.909 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.admin.41148.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74157.log.gz 2024-09-16T11:53:59.918 INFO:teuthology.orchestra.run.smithi103.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.41148.log.gz 2024-09-16T11:53:59.918 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50636.log 2024-09-16T11:53:59.919 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52298.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52298.log.gz 2024-09-16T11:53:59.933 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47603.log 2024-09-16T11:53:59.933 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50636.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50636.log.gz 2024-09-16T11:53:59.945 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46074.log 2024-09-16T11:53:59.946 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.47603.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47603.log.gz 2024-09-16T11:53:59.963 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67758.log 2024-09-16T11:53:59.968 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46074.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46074.log.gz 2024-09-16T11:53:59.972 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70392.log 2024-09-16T11:53:59.985 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67758.log: 88.0% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48576.log.gz 2024-09-16T11:53:59.985 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67758.log.gz 2024-09-16T11:53:59.996 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71364.log 2024-09-16T11:53:59.996 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70392.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51856.log.gz 2024-09-16T11:53:59.996 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70392.log.gz 2024-09-16T11:53:59.996 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70977.log 2024-09-16T11:53:59.996 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.71364.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71364.log.gz 2024-09-16T11:54:00.003 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43648.log 2024-09-16T11:54:00.003 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70977.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70977.log.gz 2024-09-16T11:54:00.004 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63790.log 2024-09-16T11:54:00.010 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.43648.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43648.log.gz 2024-09-16T11:54:00.010 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55427.log 2024-09-16T11:54:00.013 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.63790.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63790.log.gz 2024-09-16T11:54:00.013 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38338.log 2024-09-16T11:54:00.014 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55427.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55427.log.gz 2024-09-16T11:54:00.014 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38742.log 2024-09-16T11:54:00.016 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.1.38338.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38338.log.gz 2024-09-16T11:54:00.017 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62790.log 2024-09-16T11:54:00.017 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.38742.log: 51.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38742.log.gz 2024-09-16T11:54:00.018 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55520.log 2024-09-16T11:54:00.020 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62790.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62790.log.gz 2024-09-16T11:54:00.020 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46578.log 2024-09-16T11:54:00.022 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55520.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55520.log.gz 2024-09-16T11:54:00.023 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68549.log 2024-09-16T11:54:00.024 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46578.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46578.log.gz 2024-09-16T11:54:00.024 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62466.log 2024-09-16T11:54:00.025 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68549.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68549.log.gz 2024-09-16T11:54:00.026 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50241.log 2024-09-16T11:54:00.026 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62466.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62466.log.gz 2024-09-16T11:54:00.028 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55291.log 2024-09-16T11:54:00.028 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50241.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50241.log.gz 2024-09-16T11:54:00.028 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41803.log 2024-09-16T11:54:00.030 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55291.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55291.log.gz 2024-09-16T11:54:00.030 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74736.log 2024-09-16T11:54:00.031 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.41803.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41803.log.gz 2024-09-16T11:54:00.033 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44459.log 2024-09-16T11:54:00.033 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74736.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74736.log.gz 2024-09-16T11:54:00.033 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64207.log 2024-09-16T11:54:00.035 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44459.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44459.log.gz 2024-09-16T11:54:00.035 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38271.log 2024-09-16T11:54:00.035 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.64207.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64207.log.gz 2024-09-16T11:54:00.036 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60181.log 2024-09-16T11:54:00.041 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.38271.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48339.log.gz 2024-09-16T11:54:00.045 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44389.log 2024-09-16T11:54:00.047 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.60181.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60181.log.gz 2024-09-16T11:54:00.047 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43422.log 2024-09-16T11:54:00.049 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.44389.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44389.log.gz 2024-09-16T11:54:00.050 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61233.log 2024-09-16T11:54:00.051 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.43422.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43422.log.gz 2024-09-16T11:54:00.052 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52437.log 2024-09-16T11:54:00.053 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61233.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61233.log.gz 2024-09-16T11:54:00.053 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70842.log 2024-09-16T11:54:00.067 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.52437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66101.log 2024-09-16T11:54:00.067 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70842.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70842.log.gz 2024-09-16T11:54:00.074 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56532.log 2024-09-16T11:54:00.087 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67779.log 2024-09-16T11:54:00.087 INFO:teuthology.orchestra.run.smithi103.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66101.log.gz 2024-09-16T11:54:00.087 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.56532.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56532.log.gz 2024-09-16T11:54:00.091 INFO:teuthology.orchestra.run.smithi103.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52437.log.gz 2024-09-16T11:54:00.092 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62418.log 2024-09-16T11:54:00.093 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.67779.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67779.log.gz 2024-09-16T11:54:00.094 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48893.log 2024-09-16T11:54:00.094 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.62418.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62418.log.gz 2024-09-16T11:54:00.095 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52694.log 2024-09-16T11:54:00.108 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61482.log 2024-09-16T11:54:00.108 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.52694.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52694.log.gz 2024-09-16T11:54:00.123 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70527.log 2024-09-16T11:54:00.123 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61482.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61482.log.gz 2024-09-16T11:54:00.137 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65626.log 2024-09-16T11:54:00.137 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70527.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70527.log.gz 2024-09-16T11:54:00.144 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.42203.log 2024-09-16T11:54:00.144 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65626.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65626.log.gz 2024-09-16T11:54:00.153 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55986.log 2024-09-16T11:54:00.157 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.2.42203.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.42203.log.gz 2024-09-16T11:54:00.162 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53011.log 2024-09-16T11:54:00.163 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.55986.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55986.log.gz 2024-09-16T11:54:00.177 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57677.log 2024-09-16T11:54:00.177 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53011.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53011.log.gz 2024-09-16T11:54:00.191 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57653.log 2024-09-16T11:54:00.191 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57677.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57677.log.gz 2024-09-16T11:54:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42795.log 2024-09-16T11:54:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57653.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57653.log.gz 2024-09-16T11:54:00.208 INFO:teuthology.orchestra.run.smithi103.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38271.log.gz 2024-09-16T11:54:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62531.log 2024-09-16T11:54:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42795.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42795.log.gz 2024-09-16T11:54:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66618.log 2024-09-16T11:54:00.215 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.62531.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51777.log.gz 2024-09-16T11:54:00.223 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62531.log.gz 2024-09-16T11:54:00.223 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49326.log 2024-09-16T11:54:00.227 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66618.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66618.log.gz 2024-09-16T11:54:00.227 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56287.log 2024-09-16T11:54:00.227 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.49326.log: --verbose -- /var/log/ceph/cluster2-client.mirror.48418.log 2024-09-16T11:54:00.228 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.56287.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56287.log.gz 2024-09-16T11:54:00.228 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45750.log 2024-09-16T11:54:00.229 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48418.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51584.log.gz 2024-09-16T11:54:00.239 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68298.log 2024-09-16T11:54:00.239 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52182.log.gz 2024-09-16T11:54:00.244 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.45750.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45750.log.gz 2024-09-16T11:54:00.246 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45472.log 2024-09-16T11:54:00.246 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40220.log 2024-09-16T11:54:00.246 INFO:teuthology.orchestra.run.smithi103.stderr: 88.1%/var/log/ceph/cluster1-client.mirror.45472.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68298.log.gz 2024-09-16T11:54:00.247 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53457.log 2024-09-16T11:54:00.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.40220.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40220.log.gz 2024-09-16T11:54:00.249 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54690.log 2024-09-16T11:54:00.249 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53457.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53457.log.gz 2024-09-16T11:54:00.249 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65285.log 2024-09-16T11:54:00.250 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54690.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54690.log.gz 2024-09-16T11:54:00.250 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48497.log 2024-09-16T11:54:00.252 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65285.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65285.log.gz 2024-09-16T11:54:00.252 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72684.log 2024-09-16T11:54:00.252 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5%/var/log/ceph/cluster2-client.mirror.48497.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45472.log.gz 2024-09-16T11:54:00.266 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69740.log 2024-09-16T11:54:00.268 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.72684.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72684.log.gz 2024-09-16T11:54:00.268 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70550.log 2024-09-16T11:54:00.269 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69740.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69740.log.gz 2024-09-16T11:54:00.269 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55849.log 2024-09-16T11:54:00.270 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70550.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70550.log.gz 2024-09-16T11:54:00.270 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68952.log 2024-09-16T11:54:00.271 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55849.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55849.log.gz 2024-09-16T11:54:00.271 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74576.log 2024-09-16T11:54:00.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.68952.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68952.log.gz 2024-09-16T11:54:00.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58823.log 2024-09-16T11:54:00.273 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.admin.74576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74576.log.gz 2024-09-16T11:54:00.273 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50836.log 2024-09-16T11:54:00.274 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.58823.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58823.log.gz 2024-09-16T11:54:00.274 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55829.log 2024-09-16T11:54:00.287 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.50836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.58272.log 2024-09-16T11:54:00.289 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.55829.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55829.log.gz 2024-09-16T11:54:00.296 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60144.log 2024-09-16T11:54:00.309 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.0.58272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57926.log 2024-09-16T11:54:00.310 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.60144.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60144.log.gz 2024-09-16T11:54:00.316 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.60870.log 2024-09-16T11:54:00.316 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.57926.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57926.log.gz 2024-09-16T11:54:00.325 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57971.log 2024-09-16T11:54:00.331 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.3.60870.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48893.log.gz 2024-09-16T11:54:00.343 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61683.log 2024-09-16T11:54:00.356 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.57971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42600.log 2024-09-16T11:54:00.357 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.61683.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61683.log.gz 2024-09-16T11:54:00.358 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57971.log.gz 2024-09-16T11:54:00.371 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53288.log 2024-09-16T11:54:00.376 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.42600.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42600.log.gz 2024-09-16T11:54:00.379 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.58272.log.gz 2024-09-16T11:54:00.381 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66659.log 2024-09-16T11:54:00.382 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.53288.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53288.log.gz 2024-09-16T11:54:00.382 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70437.log 2024-09-16T11:54:00.382 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66659.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66659.log.gz 2024-09-16T11:54:00.384 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48952.log 2024-09-16T11:54:00.384 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70437.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70437.log.gz 2024-09-16T11:54:00.385 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66452.log 2024-09-16T11:54:00.385 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48952.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48952.log.gz 2024-09-16T11:54:00.385 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68798.log 2024-09-16T11:54:00.387 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.66452.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66452.log.gz 2024-09-16T11:54:00.387 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65243.log 2024-09-16T11:54:00.388 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.68798.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68798.log.gz 2024-09-16T11:54:00.389 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69020.log 2024-09-16T11:54:00.392 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.65243.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65243.log.gz 2024-09-16T11:54:00.392 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46352.log 2024-09-16T11:54:00.392 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.69020.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69020.log.gz 2024-09-16T11:54:00.392 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46778.log 2024-09-16T11:54:00.392 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46352.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46352.log.gz 2024-09-16T11:54:00.394 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70797.log 2024-09-16T11:54:00.395 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.46778.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46778.log.gz 2024-09-16T11:54:00.395 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54060.log 2024-09-16T11:54:00.395 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.70797.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70797.log.gz 2024-09-16T11:54:00.395 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48238.log 2024-09-16T11:54:00.396 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.54060.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54060.log.gz 2024-09-16T11:54:00.397 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61530.log 2024-09-16T11:54:00.397 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster2-client.mirror.48238.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48238.log.gz 2024-09-16T11:54:00.413 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cluster1-client.mirror.61530.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61530.log.gz 2024-09-16T11:54:00.447 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48497.log.gz 2024-09-16T11:54:00.506 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50836.log.gz 2024-09-16T11:54:00.540 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.60870.log.gz 2024-09-16T11:54:00.621 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49326.log.gz 2024-09-16T11:54:00.675 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48418.log.gz 2024-09-16T11:54:02.730 INFO:teuthology.orchestra.run.smithi001.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-16T11:54:04.107 INFO:teuthology.orchestra.run.smithi001.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-16T11:54:04.115 INFO:teuthology.orchestra.run.smithi001.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-16T11:54:04.118 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-09-16T11:54:04.118 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m8.376s 2024-09-16T11:54:04.118 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m24.401s 2024-09-16T11:54:04.118 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.853s 2024-09-16T11:54:43.833 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-16T11:54:45.432 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-16T11:54:57.437 INFO:teuthology.orchestra.run.smithi103.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-16T11:54:57.438 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-09-16T11:54:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:real 1m1.661s 2024-09-16T11:54:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:user 2m54.609s 2024-09-16T11:54:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m7.518s 2024-09-16T11:54:57.439 INFO:tasks.ceph:Archiving logs... 2024-09-16T11:54:57.440 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/remote/smithi001/log 2024-09-16T11:54:57.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T11:55:00.764 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/remote/smithi103/log 2024-09-16T11:55:00.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T11:55:26.328 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T11:55:26.343 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T11:55:26.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:55:26.346 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:55:26.422 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T11:55:26.434 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T11:55:26.434 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:55:26.437 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:55:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:55:26.453 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:55:26.456 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:55:26.457 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-09-16T11:55:26.457 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 -452us[ -440us] +/- 45ms 2024-09-16T11:55:26.457 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 +1268us[+1279us] +/- 71ms 2024-09-16T11:55:26.457 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -585us[ -574us] +/- 73ms 2024-09-16T11:55:26.457 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:55:26.458 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:55:26.458 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-16T11:55:26.459 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 -461us[ -460us] +/- 45ms 2024-09-16T11:55:26.459 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 +1257us[+1257us] +/- 71ms 2024-09-16T11:55:26.459 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -602us[ -602us] +/- 73ms 2024-09-16T11:55:26.459 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:55:26.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T11:55:26.473 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T11:55:26.474 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T11:55:26.485 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T11:55:26.570 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T11:55:26.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:55:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:55:26.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:55:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:55:26.748 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T11:55:26.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T11:55:26.771 INFO:teuthology.task.internal:Duration was 2819.902651 seconds 2024-09-16T11:55:26.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T11:55:26.782 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T11:55:26.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:55:26.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:55:26.825 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:55:26.828 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:55:27.128 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T11:55:27.128 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-09-16T11:55:27.128 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:55:27.151 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-09-16T11:55:27.151 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:55:27.173 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T11:55:27.173 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:55:27.194 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:55:27.241 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T11:55:27.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:55:27.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:55:27.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T11:55:27.398 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T11:55:27.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:55:27.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:55:27.448 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T11:55:27.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:55:27.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:55:27.488 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-09-16T11:55:27.511 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-09-16T11:55:27.525 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:55:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:55:27.560 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:55:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:55:27.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T11:55:27.591 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T11:55:27.592 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/remote/smithi001 2024-09-16T11:55:27.592 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:55:27.634 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907167/remote/smithi103 2024-09-16T11:55:27.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:55:27.667 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T11:55:27.667 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:55:27.672 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:55:27.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T11:55:27.731 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T11:55:27.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T11:55:27.744 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T11:55:27.744 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:55:27.746 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:55:27.759 INFO:teuthology.orchestra.run.smithi001.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:55 /home/ubuntu/cephtest 2024-09-16T11:55:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:55 /home/ubuntu/cephtest 2024-09-16T11:55:27.773 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T11:55:27.786 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T11:55:27.813 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 2819.902651309967 flavor: default owner: scheduled_sangadi@teuthology success: true 2024-09-16T11:55:27.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T11:55:27.887 INFO:teuthology.run:pass